2023-07-12 19:03:51,301 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures timeout: 13 mins 2023-07-12 19:03:51,543 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6 2023-07-12 19:03:51,573 INFO [Time-limited test] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=10, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=331, ProcessCount=169, AvailableMemoryMB=9391 2023-07-12 19:03:51,580 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:03:51,581 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b, deleteOnExit=true 2023-07-12 19:03:51,581 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:03:51,581 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/test.cache.data in system properties and HBase conf 2023-07-12 19:03:51,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:03:51,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:03:51,582 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:03:51,583 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:03:51,583 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:03:51,703 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-12 19:03:52,142 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:03:52,147 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:03:52,148 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:03:52,148 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:03:52,149 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:03:52,149 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:03:52,150 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:03:52,150 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:03:52,150 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:03:52,151 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:03:52,151 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:03:52,151 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:03:52,152 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:03:52,152 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:03:52,152 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:03:52,676 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:03:52,681 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:03:52,967 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-12 19:03:53,146 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-12 19:03:53,165 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:03:53,201 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:03:53,233 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/java.io.tmpdir/Jetty_localhost_33591_hdfs____.9uyyzt/webapp 2023-07-12 19:03:53,361 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33591 2023-07-12 19:03:53,383 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:03:53,383 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:03:53,894 WARN [Listener at localhost/41637] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:03:53,966 WARN [Listener at localhost/41637] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:03:53,984 WARN [Listener at localhost/41637] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:03:53,989 INFO [Listener at localhost/41637] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:03:53,994 INFO [Listener at localhost/41637] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/java.io.tmpdir/Jetty_localhost_37883_datanode____.ld39e2/webapp 2023-07-12 19:03:54,089 INFO [Listener at localhost/41637] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37883 2023-07-12 19:03:54,397 WARN [Listener at localhost/35631] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:03:54,824 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xebdc50d57be732ce: Processing first storage report for DS-d2700b60-234b-4a38-85ec-1509dd243018 from datanode 67507a56-8e55-4294-9a03-1561e177d4f6 2023-07-12 19:03:54,826 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xebdc50d57be732ce: from storage DS-d2700b60-234b-4a38-85ec-1509dd243018 node DatanodeRegistration(127.0.0.1:46823, datanodeUuid=67507a56-8e55-4294-9a03-1561e177d4f6, infoPort=42059, infoSecurePort=0, ipcPort=35631, storageInfo=lv=-57;cid=testClusterID;nsid=688562788;c=1689188632762), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-12 19:03:54,826 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xebdc50d57be732ce: Processing first storage report for DS-b1a85401-362b-4041-b1f0-b9ae73854be2 from datanode 67507a56-8e55-4294-9a03-1561e177d4f6 2023-07-12 19:03:54,826 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xebdc50d57be732ce: from storage DS-b1a85401-362b-4041-b1f0-b9ae73854be2 node DatanodeRegistration(127.0.0.1:46823, datanodeUuid=67507a56-8e55-4294-9a03-1561e177d4f6, infoPort=42059, infoSecurePort=0, ipcPort=35631, storageInfo=lv=-57;cid=testClusterID;nsid=688562788;c=1689188632762), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:03:54,909 DEBUG [Listener at localhost/35631] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6 2023-07-12 19:03:54,988 INFO [Listener at localhost/35631] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b/zookeeper_0, clientPort=52458, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:03:55,001 INFO [Listener at localhost/35631] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52458 2023-07-12 19:03:55,013 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:55,015 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:55,672 INFO [Listener at localhost/35631] util.FSUtils(471): Created version file at hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3 with version=8 2023-07-12 19:03:55,672 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase-staging 2023-07-12 19:03:55,992 INFO [Listener at localhost/35631] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-12 19:03:56,473 INFO [Listener at localhost/35631] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:03:56,506 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,506 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,507 INFO [Listener at localhost/35631] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:03:56,507 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,507 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:03:56,648 INFO [Listener at localhost/35631] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:03:56,719 DEBUG [Listener at localhost/35631] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-12 19:03:56,813 INFO [Listener at localhost/35631] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42517 2023-07-12 19:03:56,824 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:56,826 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:56,846 INFO [Listener at localhost/35631] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42517 connecting to ZooKeeper ensemble=127.0.0.1:52458 2023-07-12 19:03:56,890 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:425170x0, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:03:56,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42517-0x1015aeb945e0000 connected 2023-07-12 19:03:56,921 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:03:56,921 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:03:56,925 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:03:56,934 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42517 2023-07-12 19:03:56,934 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42517 2023-07-12 19:03:56,934 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42517 2023-07-12 19:03:56,935 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42517 2023-07-12 19:03:56,935 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42517 2023-07-12 19:03:56,941 INFO [Listener at localhost/35631] master.HMaster(444): hbase.rootdir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3, hbase.cluster.distributed=false 2023-07-12 19:03:56,987 INFO [Listener at localhost/35631] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:03:56,987 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,988 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,988 INFO [Listener at localhost/35631] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:03:56,988 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:56,988 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:03:56,988 INFO [Listener at localhost/35631] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:03:56,990 INFO [Listener at localhost/35631] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40781 2023-07-12 19:03:56,990 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:56,992 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:56,994 INFO [Listener at localhost/35631] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40781 connecting to ZooKeeper ensemble=127.0.0.1:52458 2023-07-12 19:03:56,998 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:407810x0, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:03:56,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40781-0x1015aeb945e0001 connected 2023-07-12 19:03:57,006 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:03:57,007 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:03:57,008 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:03:57,008 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40781 2023-07-12 19:03:57,009 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40781 2023-07-12 19:03:57,009 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40781 2023-07-12 19:03:57,010 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40781 2023-07-12 19:03:57,010 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40781 2023-07-12 19:03:57,010 INFO [Listener at localhost/35631] master.HMaster(444): hbase.rootdir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3, hbase.cluster.distributed=false 2023-07-12 19:03:57,044 INFO [Listener at localhost/35631] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:03:57,044 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:57,044 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:57,044 INFO [Listener at localhost/35631] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:03:57,045 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:03:57,045 INFO [Listener at localhost/35631] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:03:57,050 INFO [Listener at localhost/35631] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:03:57,053 INFO [Listener at localhost/35631] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35493 2023-07-12 19:03:57,055 INFO [Listener at localhost/35631] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:03:57,062 DEBUG [Listener at localhost/35631] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:03:57,062 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:57,064 INFO [Listener at localhost/35631] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:57,066 INFO [Listener at localhost/35631] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35493 connecting to ZooKeeper ensemble=127.0.0.1:52458 2023-07-12 19:03:57,071 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:354930x0, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:03:57,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:35493-0x1015aeb945e0002 connected 2023-07-12 19:03:57,073 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:03:57,073 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:03:57,074 DEBUG [Listener at localhost/35631] zookeeper.ZKUtil(164): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:03:57,075 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35493 2023-07-12 19:03:57,075 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35493 2023-07-12 19:03:57,075 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35493 2023-07-12 19:03:57,076 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35493 2023-07-12 19:03:57,076 DEBUG [Listener at localhost/35631] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35493 2023-07-12 19:03:57,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:03:57,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:57,089 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:03:57,089 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:03:57,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:57,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:03:57,109 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:03:57,109 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:03:57,109 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:03:57,109 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:57,109 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:57,110 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:03:57,110 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:03:57,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42517,1689188635819 from backup master directory 2023-07-12 19:03:57,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:03:57,112 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase4.apache.org,42517,1689188635819; waiting to become the next active master 2023-07-12 19:03:57,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:03:57,115 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:57,115 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:03:57,115 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:03:57,116 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:03:57,116 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:57,119 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-12 19:03:57,120 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-12 19:03:57,204 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase.id with ID: 1f8d064b-8a1f-4259-b420-c3d7082434f9 2023-07-12 19:03:57,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:03:57,260 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:57,260 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:57,297 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x37cf94bc to 127.0.0.1:52458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:03:57,329 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ac8ad09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:03:57,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:03:57,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:03:57,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-12 19:03:57,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-12 19:03:57,371 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:03:57,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:03:57,376 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:03:57,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store-tmp 2023-07-12 19:03:57,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:03:57,838 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:03:57,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:03:57,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:03:57,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:03:57,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:03:57,839 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:03:57,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:03:57,841 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:57,866 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42517%2C1689188635819, suffix=, logDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819, archiveDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/oldWALs, maxLogs=10 2023-07-12 19:03:57,944 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK] 2023-07-12 19:03:57,952 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:03:58,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:03:58,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK]] 2023-07-12 19:03:58,020 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:03:58,020 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:03:58,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,026 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,101 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,109 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:03:58,133 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:03:58,147 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:58,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,172 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:03:58,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:03:58,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60553449, jitterRate=-0.09768329560756683}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:03:58,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:03:58,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:03:58,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:03:58,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:03:58,199 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:03:58,201 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 19:03:58,234 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-07-12 19:03:58,234 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:03:58,260 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:03:58,266 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:03:58,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:03:58,295 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:03:58,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:03:58,301 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:03:58,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:03:58,308 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,308 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,309 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:03:58,310 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:03:58,321 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:03:58,325 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:03:58,325 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:03:58,325 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:03:58,325 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,325 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,325 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42517,1689188635819, sessionid=0x1015aeb945e0000, setting cluster-up flag (Was=false) 2023-07-12 19:03:58,329 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegionServer(951): ClusterId : 1f8d064b-8a1f-4259-b420-c3d7082434f9 2023-07-12 19:03:58,341 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,341 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,349 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:03:58,350 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:58,355 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,355 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:58,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:03:58,363 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:03:58,365 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.hbase-snapshot/.tmp 2023-07-12 19:03:58,381 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(951): ClusterId : 1f8d064b-8a1f-4259-b420-c3d7082434f9 2023-07-12 19:03:58,386 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:03:58,391 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:03:58,392 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:03:58,395 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:03:58,396 DEBUG [RS:0;jenkins-hbase4:35493] zookeeper.ReadOnlyZKClient(139): Connect 0x6a3d4524 to 127.0.0.1:52458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:03:58,402 DEBUG [RS:0;jenkins-hbase4:35493] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64faa613, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:03:58,402 DEBUG [RS:0;jenkins-hbase4:35493] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60c52e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:03:58,435 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35493 2023-07-12 19:03:58,439 INFO [RS:0;jenkins-hbase4:35493] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:03:58,439 INFO [RS:0;jenkins-hbase4:35493] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:03:58,439 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:03:58,441 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42517,1689188635819 with isa=jenkins-hbase4.apache.org/172.31.14.131:35493, startcode=1689188637043 2023-07-12 19:03:58,462 DEBUG [RS:0;jenkins-hbase4:35493] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:03:58,497 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:03:58,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:03:58,507 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:03:58,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188668509 2023-07-12 19:03:58,512 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:03:58,514 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:03:58,514 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:03:58,520 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:03:58,525 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43677, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:03:58,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:03:58,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:03:58,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:03:58,534 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42517] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:03:58,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:03:58,535 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:03:58,535 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:03:58,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:03:58,538 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:03:58,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:03:58,541 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:03:58,543 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188638543,5,FailOnTimeoutGroup] 2023-07-12 19:03:58,545 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188638544,5,FailOnTimeoutGroup] 2023-07-12 19:03:58,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:03:58,548 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,565 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(2830): Master is not running yet 2023-07-12 19:03:58,566 WARN [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-12 19:03:58,667 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,42517,1689188635819 with isa=jenkins-hbase4.apache.org/172.31.14.131:35493, startcode=1689188637043 2023-07-12 19:03:58,673 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42517] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,678 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3 2023-07-12 19:03:58,678 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:41637 2023-07-12 19:03:58,678 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:03:58,683 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:03:58,684 DEBUG [RS:0;jenkins-hbase4:35493] zookeeper.ZKUtil(162): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,684 WARN [RS:0;jenkins-hbase4:35493] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:03:58,684 INFO [RS:0;jenkins-hbase4:35493] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:03:58,685 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1948): logDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/WALs/jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,686 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35493,1689188637043] 2023-07-12 19:03:58,693 DEBUG [RS:0;jenkins-hbase4:35493] zookeeper.ZKUtil(162): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,702 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:03:58,710 INFO [RS:0;jenkins-hbase4:35493] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:03:58,728 INFO [RS:0;jenkins-hbase4:35493] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:03:58,732 INFO [RS:0;jenkins-hbase4:35493] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:03:58,732 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,733 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:03:58,739 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,740 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,740 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,740 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,740 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,740 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,741 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:03:58,741 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,741 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,741 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,741 DEBUG [RS:0;jenkins-hbase4:35493] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:03:58,742 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,742 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,742 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,757 INFO [RS:0;jenkins-hbase4:35493] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:03:58,759 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35493,1689188637043-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:03:58,774 INFO [RS:0;jenkins-hbase4:35493] regionserver.Replication(203): jenkins-hbase4.apache.org,35493,1689188637043 started 2023-07-12 19:03:58,775 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,35493,1689188637043, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35493, sessionid=0x1015aeb945e0002 2023-07-12 19:03:58,775 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:03:58,775 DEBUG [RS:0;jenkins-hbase4:35493] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,775 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35493,1689188637043' 2023-07-12 19:03:58,775 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35493,1689188637043' 2023-07-12 19:03:58,776 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:03:58,777 DEBUG [RS:0;jenkins-hbase4:35493] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:03:58,777 DEBUG [RS:0;jenkins-hbase4:35493] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:03:58,777 INFO [RS:0;jenkins-hbase4:35493] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 19:03:58,777 INFO [RS:0;jenkins-hbase4:35493] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 19:03:58,889 INFO [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35493%2C1689188637043, suffix=, logDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/WALs/jenkins-hbase4.apache.org,35493,1689188637043, archiveDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/oldWALs, maxLogs=32 2023-07-12 19:03:58,909 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK] 2023-07-12 19:03:58,912 INFO [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/WALs/jenkins-hbase4.apache.org,35493,1689188637043/jenkins-hbase4.apache.org%2C35493%2C1689188637043.1689188638892 2023-07-12 19:03:58,913 DEBUG [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK]] 2023-07-12 19:03:58,955 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:03:58,956 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:03:58,956 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3 2023-07-12 19:03:59,370 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:03:59,373 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:03:59,376 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/info 2023-07-12 19:03:59,377 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:03:59,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,378 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:03:59,381 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:03:59,381 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:03:59,382 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,382 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:03:59,384 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/table 2023-07-12 19:03:59,385 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:03:59,386 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,387 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740 2023-07-12 19:03:59,388 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740 2023-07-12 19:03:59,392 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:03:59,394 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:03:59,398 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:03:59,399 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65687026, jitterRate=-0.021187037229537964}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:03:59,399 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:03:59,399 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:03:59,399 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:03:59,399 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:03:59,400 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:03:59,400 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:03:59,400 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:03:59,401 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:03:59,406 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:03:59,406 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:03:59,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:03:59,428 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:03:59,430 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:03:59,582 DEBUG [jenkins-hbase4:42517] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:03:59,585 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35493,1689188637043, state=OPENING 2023-07-12 19:03:59,591 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:03:59,593 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:59,593 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:03:59,594 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:03:59,594 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:03:59,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:03:59,778 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:03:59,780 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:03:59,784 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49596, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:03:59,794 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:03:59,795 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:03:59,798 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35493%2C1689188637043.meta, suffix=.meta, logDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/WALs/jenkins-hbase4.apache.org,35493,1689188637043, archiveDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/oldWALs, maxLogs=32 2023-07-12 19:03:59,814 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK] 2023-07-12 19:03:59,819 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/WALs/jenkins-hbase4.apache.org,35493,1689188637043/jenkins-hbase4.apache.org%2C35493%2C1689188637043.meta.1689188639799.meta 2023-07-12 19:03:59,820 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK]] 2023-07-12 19:03:59,820 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:03:59,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:03:59,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:03:59,845 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:03:59,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:03:59,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:03:59,853 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:03:59,853 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:03:59,857 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:03:59,859 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/info 2023-07-12 19:03:59,859 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/info 2023-07-12 19:03:59,860 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:03:59,861 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,861 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:03:59,862 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:03:59,862 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:03:59,863 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:03:59,864 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,864 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:03:59,866 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/table 2023-07-12 19:03:59,866 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/table 2023-07-12 19:03:59,866 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:03:59,867 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:03:59,869 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740 2023-07-12 19:03:59,872 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740 2023-07-12 19:03:59,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:03:59,877 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:03:59,879 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72196599, jitterRate=0.0758131593465805}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:03:59,879 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:03:59,888 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689188639770 2023-07-12 19:03:59,905 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:03:59,906 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:03:59,906 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35493,1689188637043, state=OPEN 2023-07-12 19:03:59,912 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:03:59,912 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:03:59,912 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:03:59,912 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:03:59,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:03:59,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35493,1689188637043 in 316 msec 2023-07-12 19:03:59,921 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:03:59,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 503 msec 2023-07-12 19:03:59,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5040 sec 2023-07-12 19:03:59,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188639927, completionTime=-1 2023-07-12 19:03:59,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:03:59,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:03:59,994 DEBUG [hconnection-0x1c0725f5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:03:59,998 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:00,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:00,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188700015 2023-07-12 19:04:00,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188760015 2023-07-12 19:04:00,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-07-12 19:04:00,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1689188635819-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:00,037 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1689188635819-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:00,038 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1689188635819-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:00,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42517, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:00,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:00,046 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:00,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:04:00,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:00,064 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:00,066 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:00,068 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:00,089 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,091 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82 empty. 2023-07-12 19:04:00,092 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,092 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:04:00,112 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:00,114 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => aa0f24b593630b6804492ebe1afc1d82, NAME => 'hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp 2023-07-12 19:04:00,526 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:00,526 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing aa0f24b593630b6804492ebe1afc1d82, disabling compactions & flushes 2023-07-12 19:04:00,527 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. after waiting 0 ms 2023-07-12 19:04:00,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,527 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,527 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for aa0f24b593630b6804492ebe1afc1d82: 2023-07-12 19:04:00,531 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:00,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188640534"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188640534"}]},"ts":"1689188640534"} 2023-07-12 19:04:00,572 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:00,574 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:00,579 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188640574"}]},"ts":"1689188640574"} 2023-07-12 19:04:00,583 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:04:00,590 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=aa0f24b593630b6804492ebe1afc1d82, ASSIGN}] 2023-07-12 19:04:00,594 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=aa0f24b593630b6804492ebe1afc1d82, ASSIGN 2023-07-12 19:04:00,595 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=aa0f24b593630b6804492ebe1afc1d82, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35493,1689188637043; forceNewPlan=false, retain=false 2023-07-12 19:04:00,747 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=aa0f24b593630b6804492ebe1afc1d82, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:00,747 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188640747"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188640747"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188640747"}]},"ts":"1689188640747"} 2023-07-12 19:04:00,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure aa0f24b593630b6804492ebe1afc1d82, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:00,914 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,914 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => aa0f24b593630b6804492ebe1afc1d82, NAME => 'hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:00,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:00,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,915 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,917 INFO [StoreOpener-aa0f24b593630b6804492ebe1afc1d82-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,919 DEBUG [StoreOpener-aa0f24b593630b6804492ebe1afc1d82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/info 2023-07-12 19:04:00,919 DEBUG [StoreOpener-aa0f24b593630b6804492ebe1afc1d82-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/info 2023-07-12 19:04:00,920 INFO [StoreOpener-aa0f24b593630b6804492ebe1afc1d82-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region aa0f24b593630b6804492ebe1afc1d82 columnFamilyName info 2023-07-12 19:04:00,921 INFO [StoreOpener-aa0f24b593630b6804492ebe1afc1d82-1] regionserver.HStore(310): Store=aa0f24b593630b6804492ebe1afc1d82/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:00,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,928 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:00,931 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:00,933 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened aa0f24b593630b6804492ebe1afc1d82; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61444842, jitterRate=-0.08440050482749939}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:00,933 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for aa0f24b593630b6804492ebe1afc1d82: 2023-07-12 19:04:00,934 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82., pid=6, masterSystemTime=1689188640908 2023-07-12 19:04:00,938 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,938 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:00,939 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=aa0f24b593630b6804492ebe1afc1d82, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:00,940 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188640939"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188640939"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188640939"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188640939"}]},"ts":"1689188640939"} 2023-07-12 19:04:00,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:04:00,946 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure aa0f24b593630b6804492ebe1afc1d82, server=jenkins-hbase4.apache.org,35493,1689188637043 in 188 msec 2023-07-12 19:04:00,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:04:00,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=aa0f24b593630b6804492ebe1afc1d82, ASSIGN in 356 msec 2023-07-12 19:04:00,951 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:00,951 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188640951"}]},"ts":"1689188640951"} 2023-07-12 19:04:00,954 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:04:00,957 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:00,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 901 msec 2023-07-12 19:04:00,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:04:00,968 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:00,968 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:00,968 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:01,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:01,033 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:01,041 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 40 msec 2023-07-12 19:04:01,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:01,059 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:01,065 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 19 msec 2023-07-12 19:04:01,080 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:01,082 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:01,082 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.966sec 2023-07-12 19:04:01,084 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:01,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:01,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:01,087 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1689188635819-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:01,088 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1689188635819-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:01,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:01,187 DEBUG [Listener at localhost/35631] zookeeper.ReadOnlyZKClient(139): Connect 0x66452f5c to 127.0.0.1:52458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:01,192 DEBUG [Listener at localhost/35631] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a154fb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:01,205 DEBUG [hconnection-0x64d6ff4d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:01,216 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:01,225 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:04:01,227 WARN [Listener at localhost/35631] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:01,231 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:01,234 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:01,236 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:01,243 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:01,243 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:01,243 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:01,243 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:01,244 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08 empty. 2023-07-12 19:04:01,244 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f empty. 2023-07-12 19:04:01,244 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e empty. 2023-07-12 19:04:01,246 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0 empty. 2023-07-12 19:04:01,246 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:01,247 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:01,247 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:01,247 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:01,247 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDisableTableWithFailoverAtStep3 regions 2023-07-12 19:04:01,266 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp/data/default/testDisableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:01,272 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 3191b1aba876ba812e8996d236df80e0, NAME => 'testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp 2023-07-12 19:04:01,272 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => c5a27bc10f09432eeb523bc4845cad08, NAME => 'testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp 2023-07-12 19:04:01,280 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => a80cb2429575b7f0836e01db5c81422f, NAME => 'testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp 2023-07-12 19:04:01,315 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:01,316 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing 3191b1aba876ba812e8996d236df80e0, disabling compactions & flushes 2023-07-12 19:04:01,316 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:01,316 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:01,316 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. after waiting 0 ms 2023-07-12 19:04:01,316 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:01,316 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:01,316 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for 3191b1aba876ba812e8996d236df80e0: 2023-07-12 19:04:01,317 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => a16b7ac8e79316dd09f4513dea13b25e, NAME => 'testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testDisableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.tmp 2023-07-12 19:04:01,317 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:01,317 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing a80cb2429575b7f0836e01db5c81422f, disabling compactions & flushes 2023-07-12 19:04:01,317 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:01,318 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:01,318 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. after waiting 0 ms 2023-07-12 19:04:01,318 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:01,318 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:01,318 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for a80cb2429575b7f0836e01db5c81422f: 2023-07-12 19:04:01,331 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:01,331 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing a16b7ac8e79316dd09f4513dea13b25e, disabling compactions & flushes 2023-07-12 19:04:01,331 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:01,331 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:01,332 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. after waiting 0 ms 2023-07-12 19:04:01,332 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:01,332 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:01,332 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for a16b7ac8e79316dd09f4513dea13b25e: 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing c5a27bc10f09432eeb523bc4845cad08, disabling compactions & flushes 2023-07-12 19:04:01,714 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. after waiting 0 ms 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:01,714 INFO [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:01,714 DEBUG [RegionOpenAndInit-testDisableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for c5a27bc10f09432eeb523bc4845cad08: 2023-07-12 19:04:01,719 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:01,720 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188641720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188641720"}]},"ts":"1689188641720"} 2023-07-12 19:04:01,720 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188641720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188641720"}]},"ts":"1689188641720"} 2023-07-12 19:04:01,721 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188641720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188641720"}]},"ts":"1689188641720"} 2023-07-12 19:04:01,721 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188641720"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188641720"}]},"ts":"1689188641720"} 2023-07-12 19:04:01,765 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 19:04:01,766 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:01,766 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188641766"}]},"ts":"1689188641766"} 2023-07-12 19:04:01,769 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:01,774 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, ASSIGN}] 2023-07-12 19:04:01,777 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, ASSIGN 2023-07-12 19:04:01,778 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35493,1689188637043; forceNewPlan=false, retain=false 2023-07-12 19:04:01,779 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, ASSIGN 2023-07-12 19:04:01,780 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35493,1689188637043; forceNewPlan=false, retain=false 2023-07-12 19:04:01,781 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, ASSIGN 2023-07-12 19:04:01,782 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35493,1689188637043; forceNewPlan=false, retain=false 2023-07-12 19:04:01,783 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, ASSIGN 2023-07-12 19:04:01,784 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35493,1689188637043; forceNewPlan=false, retain=false 2023-07-12 19:04:01,929 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=a80cb2429575b7f0836e01db5c81422f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:01,930 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188641929"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188641929"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188641929"}]},"ts":"1689188641929"} 2023-07-12 19:04:01,933 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE; OpenRegionProcedure a80cb2429575b7f0836e01db5c81422f, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:01,938 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:01,938 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188641938"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188641938"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188641938"}]},"ts":"1689188641938"} 2023-07-12 19:04:01,940 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=13, state=RUNNABLE; OpenRegionProcedure a16b7ac8e79316dd09f4513dea13b25e, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:01,944 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c5a27bc10f09432eeb523bc4845cad08, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:01,944 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188641944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188641944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188641944"}]},"ts":"1689188641944"} 2023-07-12 19:04:01,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure c5a27bc10f09432eeb523bc4845cad08, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:01,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=3191b1aba876ba812e8996d236df80e0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:01,951 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188641950"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188641950"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188641950"}]},"ts":"1689188641950"} 2023-07-12 19:04:01,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=11, state=RUNNABLE; OpenRegionProcedure 3191b1aba876ba812e8996d236df80e0, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:02,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:02,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a80cb2429575b7f0836e01db5c81422f, NAME => 'testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 19:04:02,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:02,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,097 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,099 DEBUG [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/f1 2023-07-12 19:04:02,099 DEBUG [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/f1 2023-07-12 19:04:02,099 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a80cb2429575b7f0836e01db5c81422f columnFamilyName f1 2023-07-12 19:04:02,100 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] regionserver.HStore(310): Store=a80cb2429575b7f0836e01db5c81422f/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,100 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,102 DEBUG [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/f2 2023-07-12 19:04:02,102 DEBUG [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/f2 2023-07-12 19:04:02,103 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a80cb2429575b7f0836e01db5c81422f columnFamilyName f2 2023-07-12 19:04:02,103 INFO [StoreOpener-a80cb2429575b7f0836e01db5c81422f-1] regionserver.HStore(310): Store=a80cb2429575b7f0836e01db5c81422f/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,109 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:02,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:02,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:02,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a80cb2429575b7f0836e01db5c81422f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63554808, jitterRate=-0.052959561347961426}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:02,118 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a80cb2429575b7f0836e01db5c81422f: 2023-07-12 19:04:02,120 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f., pid=14, masterSystemTime=1689188642086 2023-07-12 19:04:02,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:02,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:02,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:02,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3191b1aba876ba812e8996d236df80e0, NAME => 'testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 19:04:02,123 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=a80cb2429575b7f0836e01db5c81422f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:02,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:02,124 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188642123"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188642123"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188642123"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188642123"}]},"ts":"1689188642123"} 2023-07-12 19:04:02,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,126 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,129 DEBUG [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/f1 2023-07-12 19:04:02,129 DEBUG [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/f1 2023-07-12 19:04:02,130 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3191b1aba876ba812e8996d236df80e0 columnFamilyName f1 2023-07-12 19:04:02,131 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] regionserver.HStore(310): Store=3191b1aba876ba812e8996d236df80e0/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,132 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=12 2023-07-12 19:04:02,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=12, state=SUCCESS; OpenRegionProcedure a80cb2429575b7f0836e01db5c81422f, server=jenkins-hbase4.apache.org,35493,1689188637043 in 194 msec 2023-07-12 19:04:02,134 DEBUG [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/f2 2023-07-12 19:04:02,134 DEBUG [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/f2 2023-07-12 19:04:02,135 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3191b1aba876ba812e8996d236df80e0 columnFamilyName f2 2023-07-12 19:04:02,136 INFO [StoreOpener-3191b1aba876ba812e8996d236df80e0-1] regionserver.HStore(310): Store=3191b1aba876ba812e8996d236df80e0/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, ASSIGN in 360 msec 2023-07-12 19:04:02,139 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,140 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:02,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:02,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:02,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3191b1aba876ba812e8996d236df80e0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70078254, jitterRate=0.04424735903739929}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:02,148 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3191b1aba876ba812e8996d236df80e0: 2023-07-12 19:04:02,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0., pid=17, masterSystemTime=1689188642086 2023-07-12 19:04:02,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:02,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:02,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:02,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c5a27bc10f09432eeb523bc4845cad08, NAME => 'testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 19:04:02,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:02,152 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=3191b1aba876ba812e8996d236df80e0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:02,152 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,153 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188642152"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188642152"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188642152"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188642152"}]},"ts":"1689188642152"} 2023-07-12 19:04:02,154 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,156 DEBUG [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/f1 2023-07-12 19:04:02,156 DEBUG [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/f1 2023-07-12 19:04:02,157 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c5a27bc10f09432eeb523bc4845cad08 columnFamilyName f1 2023-07-12 19:04:02,158 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] regionserver.HStore(310): Store=c5a27bc10f09432eeb523bc4845cad08/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,158 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=11 2023-07-12 19:04:02,159 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=11, state=SUCCESS; OpenRegionProcedure 3191b1aba876ba812e8996d236df80e0, server=jenkins-hbase4.apache.org,35493,1689188637043 in 202 msec 2023-07-12 19:04:02,160 DEBUG [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/f2 2023-07-12 19:04:02,160 DEBUG [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/f2 2023-07-12 19:04:02,161 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c5a27bc10f09432eeb523bc4845cad08 columnFamilyName f2 2023-07-12 19:04:02,161 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, ASSIGN in 385 msec 2023-07-12 19:04:02,162 INFO [StoreOpener-c5a27bc10f09432eeb523bc4845cad08-1] regionserver.HStore(310): Store=c5a27bc10f09432eeb523bc4845cad08/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,163 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,167 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:02,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:02,172 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:02,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c5a27bc10f09432eeb523bc4845cad08; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62017670, jitterRate=-0.07586470246315002}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:02,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c5a27bc10f09432eeb523bc4845cad08: 2023-07-12 19:04:02,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08., pid=16, masterSystemTime=1689188642086 2023-07-12 19:04:02,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:02,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:02,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:02,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a16b7ac8e79316dd09f4513dea13b25e, NAME => 'testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 19:04:02,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDisableTableWithFailoverAtStep3 a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,177 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c5a27bc10f09432eeb523bc4845cad08, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:02,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:02,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,177 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188642177"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188642177"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188642177"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188642177"}]},"ts":"1689188642177"} 2023-07-12 19:04:02,179 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,181 DEBUG [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/f1 2023-07-12 19:04:02,181 DEBUG [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/f1 2023-07-12 19:04:02,182 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a16b7ac8e79316dd09f4513dea13b25e columnFamilyName f1 2023-07-12 19:04:02,183 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] regionserver.HStore(310): Store=a16b7ac8e79316dd09f4513dea13b25e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,183 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-07-12 19:04:02,183 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure c5a27bc10f09432eeb523bc4845cad08, server=jenkins-hbase4.apache.org,35493,1689188637043 in 233 msec 2023-07-12 19:04:02,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, ASSIGN in 409 msec 2023-07-12 19:04:02,185 DEBUG [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/f2 2023-07-12 19:04:02,185 DEBUG [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/f2 2023-07-12 19:04:02,187 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a16b7ac8e79316dd09f4513dea13b25e columnFamilyName f2 2023-07-12 19:04:02,188 INFO [StoreOpener-a16b7ac8e79316dd09f4513dea13b25e-1] regionserver.HStore(310): Store=a16b7ac8e79316dd09f4513dea13b25e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:02,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDisableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:02,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:02,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:02,199 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a16b7ac8e79316dd09f4513dea13b25e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70097629, jitterRate=0.04453606903553009}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:02,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a16b7ac8e79316dd09f4513dea13b25e: 2023-07-12 19:04:02,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e., pid=15, masterSystemTime=1689188642086 2023-07-12 19:04:02,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:02,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:02,203 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:02,204 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188642203"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188642203"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188642203"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188642203"}]},"ts":"1689188642203"} 2023-07-12 19:04:02,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=13 2023-07-12 19:04:02,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=13, state=SUCCESS; OpenRegionProcedure a16b7ac8e79316dd09f4513dea13b25e, server=jenkins-hbase4.apache.org,35493,1689188637043 in 266 msec 2023-07-12 19:04:02,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=9 2023-07-12 19:04:02,212 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, ASSIGN in 435 msec 2023-07-12 19:04:02,213 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:02,213 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188642213"}]},"ts":"1689188642213"} 2023-07-12 19:04:02,215 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 19:04:02,218 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:02,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 in 989 msec 2023-07-12 19:04:02,232 WARN [Listener at localhost/35631] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 19:04:02,238 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,240 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:02,240 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,240 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:02,241 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:02,488 INFO [Listener at localhost/35631] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,489 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:02,490 INFO [Listener at localhost/35631] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:02,490 INFO [Listener at localhost/35631] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:02,491 ERROR [Listener at localhost/35631] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] 2023-07-12 19:04:02,492 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:02,492 INFO [Listener at localhost/35631] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:02,492 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:02,492 INFO [Listener at localhost/35631] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:02,493 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 19:04:02,496 DEBUG [Listener at localhost/35631] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[] 2023-07-12 19:04:02,497 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:02,498 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:02,498 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:02,498 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:02,499 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,499 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,499 DEBUG [Listener at localhost/35631] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:02,500 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 19:04:02,501 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:02,501 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35493,1689188637043, table=hbase:meta, region=1588230740 2023-07-12 19:04:02,502 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188672502 2023-07-12 19:04:02,503 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,503 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:02,508 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=aa0f24b593630b6804492ebe1afc1d82, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,508 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=c5a27bc10f09432eeb523bc4845cad08, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,509 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=3191b1aba876ba812e8996d236df80e0, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,509 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a80cb2429575b7f0836e01db5c81422f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,510 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,510 INFO [Listener at localhost/35631] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:02,510 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188702510 2023-07-12 19:04:02,510 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188762510 2023-07-12 19:04:02,510 INFO [Listener at localhost/35631] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 19:04:02,510 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:02,512 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:02,512 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:02,512 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:02,760 INFO [Listener at localhost/35631] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,760 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:02,760 INFO [Listener at localhost/35631] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:02,761 INFO [Listener at localhost/35631] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:02,761 ERROR [Listener at localhost/35631] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:02,761 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:02,761 INFO [Listener at localhost/35631] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:02,761 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:02,761 INFO [Listener at localhost/35631] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:02,762 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 19:04:02,764 DEBUG [Listener at localhost/35631] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0] 2023-07-12 19:04:02,765 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:02,765 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:02,766 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:02,766 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:02,766 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,766 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,767 DEBUG [Listener at localhost/35631] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:02,767 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-12 19:04:02,767 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:02,768 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35493,1689188637043, table=hbase:meta, region=1588230740 2023-07-12 19:04:02,768 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188672768 2023-07-12 19:04:02,769 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:02,769 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,773 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=aa0f24b593630b6804492ebe1afc1d82, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,774 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=c5a27bc10f09432eeb523bc4845cad08, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,774 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=3191b1aba876ba812e8996d236df80e0, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,775 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a80cb2429575b7f0836e01db5c81422f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,775 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:02,776 INFO [Listener at localhost/35631] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:02,776 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188702776 2023-07-12 19:04:02,776 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188762776 2023-07-12 19:04:02,776 INFO [Listener at localhost/35631] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 19:04:02,776 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:02,778 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188642778"}]},"ts":"1689188642778"} 2023-07-12 19:04:02,781 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 19:04:02,783 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 19:04:02,783 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:02,783 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:02,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:02,783 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:03,026 INFO [Listener at localhost/35631] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=18, state=RUNNABLE:DISABLE_TABLE_PRE_OPERATION, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:03,026 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:03,027 INFO [Listener at localhost/35631] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:03,027 INFO [Listener at localhost/35631] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:03,027 ERROR [Listener at localhost/35631] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:03,027 INFO [Listener at localhost/35631] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:03,027 INFO [Listener at localhost/35631] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:03,027 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:03,028 INFO [Listener at localhost/35631] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:03,029 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-12 19:04:03,030 DEBUG [Listener at localhost/35631] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1] 2023-07-12 19:04:03,031 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:03,031 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:03,032 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:03,032 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:03,032 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:03,033 DEBUG [Listener at localhost/35631] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:03,033 DEBUG [Listener at localhost/35631] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:03,033 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 19:04:03,034 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:03,034 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35493,1689188637043, table=hbase:meta, region=1588230740 2023-07-12 19:04:03,035 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188673035 2023-07-12 19:04:03,036 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_SET_DISABLING_TABLE_STATE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:03,036 DEBUG [Listener at localhost/35631] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:03,039 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=aa0f24b593630b6804492ebe1afc1d82, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:03,040 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=c5a27bc10f09432eeb523bc4845cad08, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:03,040 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=3191b1aba876ba812e8996d236df80e0, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:03,041 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a80cb2429575b7f0836e01db5c81422f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:03,041 DEBUG [Listener at localhost/35631] assignment.RegionStateStore(147): Load hbase:meta entry region=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:03,041 INFO [Listener at localhost/35631] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:03,042 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188703041 2023-07-12 19:04:03,042 INFO [Listener at localhost/35631] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188763042 2023-07-12 19:04:03,042 INFO [Listener at localhost/35631] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 19:04:03,042 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188643042"}]},"ts":"1689188643042"} 2023-07-12 19:04:03,044 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 19:04:03,050 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDisableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 19:04:03,050 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:03,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, UNASSIGN}] 2023-07-12 19:04:03,053 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:03,053 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:03,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:03,053 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=18, state=WAITING:DISABLE_TABLE_ADD_REPLICATION_BARRIER, locked=true; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:03,292 INFO [Listener at localhost/35631] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 19:04:03,293 INFO [Listener at localhost/35631] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:04:03,297 ERROR [Listener at localhost/35631] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 19:04:03,297 ERROR [Listener at localhost/35631] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase4.apache.org,42517,1689188635819: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:254) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDisableTableWithFailover(TestMasterFailoverWithProcedures.java:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:03,298 INFO [Listener at localhost/35631] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42517,1689188635819' ***** 2023-07-12 19:04:03,298 INFO [Listener at localhost/35631] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/35631 2023-07-12 19:04:03,299 DEBUG [M:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40bb4074, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:03,299 INFO [M:0;jenkins-hbase4:42517] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,42517,1689188635819 2023-07-12 19:04:03,300 DEBUG [M:0;jenkins-hbase4:42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37cf94bc to 127.0.0.1:52458 2023-07-12 19:04:03,300 DEBUG [M:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:03,301 INFO [M:0;jenkins-hbase4:42517] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42517,1689188635819; all regions closed. 2023-07-12 19:04:03,301 DEBUG [M:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:03,301 DEBUG [M:0;jenkins-hbase4:42517] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:03,301 DEBUG [M:0;jenkins-hbase4:42517] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:03,301 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188638544] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188638544,5,FailOnTimeoutGroup] 2023-07-12 19:04:03,302 INFO [M:0;jenkins-hbase4:42517] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:03,303 INFO [M:0;jenkins-hbase4:42517] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:03,303 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,303 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:03,303 INFO [M:0;jenkins-hbase4:42517] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:03,301 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188638543] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188638543,5,FailOnTimeoutGroup] 2023-07-12 19:04:03,301 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:03,304 DEBUG [M:0;jenkins-hbase4:42517] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:03,304 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,304 INFO [M:0;jenkins-hbase4:42517] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:03,304 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:03,304 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,305 ERROR [M:0;jenkins-hbase4:42517] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:03,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:03,305 INFO [M:0;jenkins-hbase4:42517] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 19:04:03,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:03,306 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:03,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:03,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:03,307 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 19:04:03,307 DEBUG [M:0;jenkins-hbase4:42517] zookeeper.ZKUtil(398): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:03,307 WARN [M:0;jenkins-hbase4:42517] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:03,307 INFO [M:0;jenkins-hbase4:42517] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:03,308 INFO [M:0;jenkins-hbase4:42517] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 19:04:03,314 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,314 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:03,314 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,314 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:03,314 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:03,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:03,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:03,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40781,1689188636986 from backup master directory 2023-07-12 19:04:03,321 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:03,323 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:03,323 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:03,323 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:03,323 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:03,323 DEBUG [M:0;jenkins-hbase4:42517] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:03,323 INFO [M:0;jenkins-hbase4:42517] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:03,323 DEBUG [M:0;jenkins-hbase4:42517] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:03,323 DEBUG [M:0;jenkins-hbase4:42517] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:03,324 DEBUG [M:0;jenkins-hbase4:42517] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:03,328 ERROR [M:0;jenkins-hbase4:42517] regionserver.HRegion(1825): Memstore data size is 63955 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:03,329 INFO [M:0;jenkins-hbase4:42517] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:03,329 DEBUG [M:0;jenkins-hbase4:42517] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:03,329 INFO [M:0;jenkins-hbase4:42517] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:03,329 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:03,330 INFO [M:0;jenkins-hbase4:42517] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42517 2023-07-12 19:04:03,341 DEBUG [M:0;jenkins-hbase4:42517] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42517,1689188635819 already deleted, retry=false 2023-07-12 19:04:03,357 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:03,369 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:42517 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42517 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:03,369 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:42517 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:42517 2023-07-12 19:04:03,471 DEBUG [RS:0;jenkins-hbase4:35493] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:03,474 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39335, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:03,475 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:03,506 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:03,506 INFO [M:0;jenkins-hbase4:42517] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42517,1689188635819; zookeeper connection closed. 2023-07-12 19:04:03,506 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:42517-0x1015aeb945e0000, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:03,578 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:03,681 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:03,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1cbf8713 to 127.0.0.1:52458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:03,784 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:03,789 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66b30f42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:03,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:03,789 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:03,790 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:03,799 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819 to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819-dead as it is dead 2023-07-12 19:04:03,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819-dead/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:04:03,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819-dead/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 after 4ms 2023-07-12 19:04:03,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819-dead/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:04:03,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,42517,1689188635819-dead 2023-07-12 19:04:03,807 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:03,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40781%2C1689188636986, suffix=, logDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,40781,1689188636986, archiveDir=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:03,824 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK] 2023-07-12 19:04:03,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/WALs/jenkins-hbase4.apache.org,40781,1689188636986/jenkins-hbase4.apache.org%2C40781%2C1689188636986.1689188643810 2023-07-12 19:04:03,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:46823,DS-d2700b60-234b-4a38-85ec-1509dd243018,DISK]] 2023-07-12 19:04:03,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:03,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:03,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:03,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:03,830 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:03,832 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:03,832 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:03,833 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:03,834 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 19:04:03,835 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:04:03,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 156, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=134, path=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:04:03,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=62.46 KB heapSize=74.24 KB 2023-07-12 19:04:03,888 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:03,991 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:04,094 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:04,198 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:04,302 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:04,315 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.46 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/24930159423b43f686d68698b2db65c7 2023-07-12 19:04:04,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/24930159423b43f686d68698b2db65c7 as hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/24930159423b43f686d68698b2db65c7 2023-07-12 19:04:04,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/24930159423b43f686d68698b2db65c7, entries=156, sequenceid=134, filesize=67.4 K 2023-07-12 19:04:04,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~62.46 KB/63955, heapSize ~74.23 KB/76008, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 512ms, sequenceid=134, compaction requested=false; wal=null 2023-07-12 19:04:04,374 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C42517%2C1689188635819.1689188637881 2023-07-12 19:04:04,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:04,384 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/134.seqid, newMaxSeqId=134, maxSeqId=1 2023-07-12 19:04:04,385 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=135; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65998347, jitterRate=-0.016547992825508118}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:04,386 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:04,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:04,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:04,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:04,389 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:04,390 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:04,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=18, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DisableTableProcedure) stack ids=[0, 1, 2] 2023-07-12 19:04:04,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:04,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:04,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:04,404 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:04,405 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:04,405 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:04,405 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.Procedure(966): pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:04,405 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 15 msec 2023-07-12 19:04:04,405 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:04,405 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40781] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:04,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:04,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35493,1689188637043, table=hbase:meta, region=1588230740 2023-07-12 19:04:04,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:04,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,421 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:04,421 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:04,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:04,422 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:04,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:04,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:04,424 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:04,425 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:04,425 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40781,1689188636986, sessionid=0x1015aeb945e0001, setting cluster-up flag (Was=true) 2023-07-12 19:04:04,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:04,429 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:04,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:04,433 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:04,434 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/.hbase-snapshot/.tmp 2023-07-12 19:04:04,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:04,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:04,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:04,435 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:04,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:04,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:04,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:04,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:04,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188674438 2023-07-12 19:04:04,439 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=18, state=RUNNABLE:DISABLE_TABLE_MARK_REGIONS_OFFLINE; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 2023-07-12 19:04:04,439 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:04,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:04,442 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:04,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:04,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:04,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:04,443 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188644443,5,FailOnTimeoutGroup] 2023-07-12 19:04:04,443 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188644443,5,FailOnTimeoutGroup] 2023-07-12 19:04:04,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:04,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188644444, completionTime=-1 2023-07-12 19:04:04,444 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:04,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:04,447 DEBUG [hconnection-0x7055617-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:04,449 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51628, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:04,452 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=aa0f24b593630b6804492ebe1afc1d82, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:04,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=c5a27bc10f09432eeb523bc4845cad08, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:04,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=3191b1aba876ba812e8996d236df80e0, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:04,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a80cb2429575b7f0836e01db5c81422f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:04,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=a16b7ac8e79316dd09f4513dea13b25e, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35493,1689188637043, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043, openSeqNum=2 2023-07-12 19:04:04,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:04,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188704453 2023-07-12 19:04:04,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188764453 2023-07-12 19:04:04,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 9 msec 2023-07-12 19:04:04,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40781,1689188636986-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40781,1689188636986-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40781,1689188636986-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40781, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:04,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 19:04:04,479 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:04,484 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:04,488 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:04,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.165sec 2023-07-12 19:04:04,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:04,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:04,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:04,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40781,1689188636986-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:04,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, UNASSIGN}] 2023-07-12 19:04:04,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40781,1689188636986-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:04,493 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, UNASSIGN 2023-07-12 19:04:04,494 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=a16b7ac8e79316dd09f4513dea13b25e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,495 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188644494"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188644494"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188644494"}]},"ts":"1689188644494"} 2023-07-12 19:04:04,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:04,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure a16b7ac8e79316dd09f4513dea13b25e, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:04,504 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, UNASSIGN 2023-07-12 19:04:04,505 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a80cb2429575b7f0836e01db5c81422f, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,505 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188644505"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188644505"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188644505"}]},"ts":"1689188644505"} 2023-07-12 19:04:04,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure a80cb2429575b7f0836e01db5c81422f, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:04,511 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, UNASSIGN 2023-07-12 19:04:04,512 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=3191b1aba876ba812e8996d236df80e0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,512 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188644512"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188644512"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188644512"}]},"ts":"1689188644512"} 2023-07-12 19:04:04,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure 3191b1aba876ba812e8996d236df80e0, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:04,517 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, UNASSIGN 2023-07-12 19:04:04,518 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=c5a27bc10f09432eeb523bc4845cad08, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,518 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188644518"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188644518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188644518"}]},"ts":"1689188644518"} 2023-07-12 19:04:04,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure c5a27bc10f09432eeb523bc4845cad08, server=jenkins-hbase4.apache.org,35493,1689188637043}] 2023-07-12 19:04:04,655 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,656 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:04,658 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:04,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:04,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a80cb2429575b7f0836e01db5c81422f, disabling compactions & flushes 2023-07-12 19:04:04,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:04,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:04,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. after waiting 0 ms 2023-07-12 19:04:04,660 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:04,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a80cb2429575b7f0836e01db5c81422f/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:04,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f. 2023-07-12 19:04:04,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a80cb2429575b7f0836e01db5c81422f: 2023-07-12 19:04:04,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed a80cb2429575b7f0836e01db5c81422f 2023-07-12 19:04:04,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:04,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a16b7ac8e79316dd09f4513dea13b25e, disabling compactions & flushes 2023-07-12 19:04:04,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:04,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:04,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. after waiting 0 ms 2023-07-12 19:04:04,670 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:04,671 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a80cb2429575b7f0836e01db5c81422f, regionState=CLOSED 2023-07-12 19:04:04,671 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,b,1689188641229.a80cb2429575b7f0836e01db5c81422f.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188644671"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188644671"}]},"ts":"1689188644671"} 2023-07-12 19:04:04,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/a16b7ac8e79316dd09f4513dea13b25e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:04,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 19:04:04,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure a80cb2429575b7f0836e01db5c81422f, server=jenkins-hbase4.apache.org,35493,1689188637043 in 166 msec 2023-07-12 19:04:04,676 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e. 2023-07-12 19:04:04,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a16b7ac8e79316dd09f4513dea13b25e: 2023-07-12 19:04:04,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a80cb2429575b7f0836e01db5c81422f, UNASSIGN in 189 msec 2023-07-12 19:04:04,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed a16b7ac8e79316dd09f4513dea13b25e 2023-07-12 19:04:04,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:04,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c5a27bc10f09432eeb523bc4845cad08, disabling compactions & flushes 2023-07-12 19:04:04,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:04,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:04,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. after waiting 0 ms 2023-07-12 19:04:04,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:04,680 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=a16b7ac8e79316dd09f4513dea13b25e, regionState=CLOSED 2023-07-12 19:04:04,681 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,c,1689188641229.a16b7ac8e79316dd09f4513dea13b25e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188644680"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188644680"}]},"ts":"1689188644680"} 2023-07-12 19:04:04,685 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/c5a27bc10f09432eeb523bc4845cad08/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:04,685 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 19:04:04,685 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure a16b7ac8e79316dd09f4513dea13b25e, server=jenkins-hbase4.apache.org,35493,1689188637043 in 186 msec 2023-07-12 19:04:04,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08. 2023-07-12 19:04:04,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c5a27bc10f09432eeb523bc4845cad08: 2023-07-12 19:04:04,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=a16b7ac8e79316dd09f4513dea13b25e, UNASSIGN in 198 msec 2023-07-12 19:04:04,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed c5a27bc10f09432eeb523bc4845cad08 2023-07-12 19:04:04,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:04,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3191b1aba876ba812e8996d236df80e0, disabling compactions & flushes 2023-07-12 19:04:04,689 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:04,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:04,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. after waiting 0 ms 2023-07-12 19:04:04,689 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:04,689 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=c5a27bc10f09432eeb523bc4845cad08, regionState=CLOSED 2023-07-12 19:04:04,689 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,,1689188641229.c5a27bc10f09432eeb523bc4845cad08.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188644689"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188644689"}]},"ts":"1689188644689"} 2023-07-12 19:04:04,694 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/default/testDisableTableWithFailoverAtStep3/3191b1aba876ba812e8996d236df80e0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:04,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0. 2023-07-12 19:04:04,695 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3191b1aba876ba812e8996d236df80e0: 2023-07-12 19:04:04,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 3191b1aba876ba812e8996d236df80e0 2023-07-12 19:04:04,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 19:04:04,700 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure c5a27bc10f09432eeb523bc4845cad08, server=jenkins-hbase4.apache.org,35493,1689188637043 in 171 msec 2023-07-12 19:04:04,702 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=c5a27bc10f09432eeb523bc4845cad08, UNASSIGN in 212 msec 2023-07-12 19:04:04,703 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=3191b1aba876ba812e8996d236df80e0, regionState=CLOSED 2023-07-12 19:04:04,703 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDisableTableWithFailoverAtStep3,a,1689188641229.3191b1aba876ba812e8996d236df80e0.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1689188644703"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188644703"}]},"ts":"1689188644703"} 2023-07-12 19:04:04,708 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 19:04:04,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 19:04:04,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure 3191b1aba876ba812e8996d236df80e0, server=jenkins-hbase4.apache.org,35493,1689188637043 in 191 msec 2023-07-12 19:04:04,710 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 19:04:04,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=18 2023-07-12 19:04:04,747 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testDisableTableWithFailoverAtStep3, region=3191b1aba876ba812e8996d236df80e0, UNASSIGN in 221 msec 2023-07-12 19:04:04,748 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDisableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188644748"}]},"ts":"1689188644748"} 2023-07-12 19:04:04,751 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDisableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-12 19:04:04,753 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDisableTableWithFailoverAtStep3 to state=DISABLED 2023-07-12 19:04:04,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testDisableTableWithFailoverAtStep3 in 2.5220 sec 2023-07-12 19:04:04,804 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:04:04,804 DEBUG [Listener at localhost/35631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66452f5c to 127.0.0.1:52458 2023-07-12 19:04:04,804 DEBUG [Listener at localhost/35631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:04,806 DEBUG [Listener at localhost/35631] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:04:04,806 DEBUG [Listener at localhost/35631] util.JVMClusterUtil(257): Found active master hash=594831047, stopped=false 2023-07-12 19:04:04,806 INFO [Listener at localhost/35631] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:04,809 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:04,809 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:04,809 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:04,809 INFO [Listener at localhost/35631] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:04,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:04,810 DEBUG [Listener at localhost/35631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cbf8713 to 127.0.0.1:52458 2023-07-12 19:04:04,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:04,811 DEBUG [Listener at localhost/35631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:04,812 INFO [Listener at localhost/35631] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,35493,1689188637043' ***** 2023-07-12 19:04:04,812 INFO [Listener at localhost/35631] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:04:04,812 INFO [RS:0;jenkins-hbase4:35493] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:04:04,812 INFO [RS:0;jenkins-hbase4:35493] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:04:04,813 INFO [RS:0;jenkins-hbase4:35493] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:04:04,813 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(3305): Received CLOSE for aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:04,813 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:04,814 DEBUG [RS:0;jenkins-hbase4:35493] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a3d4524 to 127.0.0.1:52458 2023-07-12 19:04:04,814 DEBUG [RS:0;jenkins-hbase4:35493] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:04,814 INFO [RS:0;jenkins-hbase4:35493] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:04:04,814 INFO [RS:0;jenkins-hbase4:35493] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:04:04,814 INFO [RS:0;jenkins-hbase4:35493] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:04:04,814 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:04:04,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing aa0f24b593630b6804492ebe1afc1d82, disabling compactions & flushes 2023-07-12 19:04:04,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:04,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:04,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. after waiting 0 ms 2023-07-12 19:04:04,819 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:04,819 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing aa0f24b593630b6804492ebe1afc1d82 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 19:04:04,824 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 19:04:04,824 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, aa0f24b593630b6804492ebe1afc1d82=hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82.} 2023-07-12 19:04:04,824 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:04:04,825 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:04,827 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1504): Waiting on 1588230740, aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:04,831 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:04,831 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:04,831 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:04,831 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:04,831 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=10.44 KB heapSize=16.86 KB 2023-07-12 19:04:04,847 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:05,031 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1504): Waiting on 1588230740, aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:05,231 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1504): Waiting on 1588230740, aa0f24b593630b6804492ebe1afc1d82 2023-07-12 19:04:05,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/.tmp/info/31a9380316e146b4af38f834b02f6129 2023-07-12 19:04:05,311 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=10.02 KB at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/.tmp/info/2e48389e881f4a8c8cc3da8c4b537b0c 2023-07-12 19:04:05,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/.tmp/info/31a9380316e146b4af38f834b02f6129 as hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/info/31a9380316e146b4af38f834b02f6129 2023-07-12 19:04:05,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/info/31a9380316e146b4af38f834b02f6129, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 19:04:05,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for aa0f24b593630b6804492ebe1afc1d82 in 516ms, sequenceid=6, compaction requested=false 2023-07-12 19:04:05,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/namespace/aa0f24b593630b6804492ebe1afc1d82/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 19:04:05,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:05,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for aa0f24b593630b6804492ebe1afc1d82: 2023-07-12 19:04:05,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689188640054.aa0f24b593630b6804492ebe1afc1d82. 2023-07-12 19:04:05,410 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-12 19:04:05,431 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:05,632 DEBUG [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:05,734 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=429 B at sequenceid=31 (bloomFilter=false), to=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/.tmp/table/8cc588c43a8549b4b90ab88da5298a3e 2023-07-12 19:04:05,743 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/.tmp/info/2e48389e881f4a8c8cc3da8c4b537b0c as hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/info/2e48389e881f4a8c8cc3da8c4b537b0c 2023-07-12 19:04:05,750 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/info/2e48389e881f4a8c8cc3da8c4b537b0c, entries=54, sequenceid=31, filesize=12.4 K 2023-07-12 19:04:05,752 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/.tmp/table/8cc588c43a8549b4b90ab88da5298a3e as hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/table/8cc588c43a8549b4b90ab88da5298a3e 2023-07-12 19:04:05,760 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/table/8cc588c43a8549b4b90ab88da5298a3e, entries=5, sequenceid=31, filesize=4.9 K 2023-07-12 19:04:05,761 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~10.44 KB/10692, heapSize ~16.58 KB/16976, currentSize=0 B/0 for 1588230740 in 930ms, sequenceid=31, compaction requested=false 2023-07-12 19:04:05,768 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/data/hbase/meta/1588230740/recovered.edits/34.seqid, newMaxSeqId=34, maxSeqId=1 2023-07-12 19:04:05,769 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:05,770 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:05,770 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:05,770 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:05,822 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 19:04:05,822 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 19:04:05,832 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,35493,1689188637043; all regions closed. 2023-07-12 19:04:05,842 DEBUG [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/oldWALs 2023-07-12 19:04:05,842 INFO [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35493%2C1689188637043.meta:.meta(num 1689188639799) 2023-07-12 19:04:05,847 DEBUG [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/oldWALs 2023-07-12 19:04:05,848 INFO [RS:0;jenkins-hbase4:35493] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35493%2C1689188637043:(num 1689188638892) 2023-07-12 19:04:05,848 DEBUG [RS:0;jenkins-hbase4:35493] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:05,848 INFO [RS:0;jenkins-hbase4:35493] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:05,848 INFO [RS:0;jenkins-hbase4:35493] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 19:04:05,848 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:05,849 INFO [RS:0;jenkins-hbase4:35493] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35493 2023-07-12 19:04:05,854 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:05,854 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35493,1689188637043 2023-07-12 19:04:05,854 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:05,857 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35493,1689188637043] 2023-07-12 19:04:05,857 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35493,1689188637043; numProcessing=1 2023-07-12 19:04:05,859 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35493,1689188637043 already deleted, retry=false 2023-07-12 19:04:05,859 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35493,1689188637043 expired; onlineServers=0 2023-07-12 19:04:05,859 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,40781,1689188636986' ***** 2023-07-12 19:04:05,859 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:04:05,859 DEBUG [M:1;jenkins-hbase4:40781] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62496cfc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:05,859 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,40781,1689188636986 2023-07-12 19:04:05,859 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,40781,1689188636986; all regions closed. 2023-07-12 19:04:05,859 DEBUG [M:1;jenkins-hbase4:40781] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:05,859 DEBUG [M:1;jenkins-hbase4:40781] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:05,859 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:05,859 DEBUG [M:1;jenkins-hbase4:40781] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188644443] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188644443,5,FailOnTimeoutGroup] 2023-07-12 19:04:05,860 INFO [M:1;jenkins-hbase4:40781] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:05,860 INFO [M:1;jenkins-hbase4:40781] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:05,859 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188644443] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188644443,5,FailOnTimeoutGroup] 2023-07-12 19:04:05,860 INFO [M:1;jenkins-hbase4:40781] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:05,860 DEBUG [M:1;jenkins-hbase4:40781] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:05,860 INFO [M:1;jenkins-hbase4:40781] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:05,861 ERROR [M:1;jenkins-hbase4:40781] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:05,861 INFO [M:1;jenkins-hbase4:40781] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:05,861 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:05,861 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:05,861 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:05,862 DEBUG [M:1;jenkins-hbase4:40781] zookeeper.ZKUtil(398): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:05,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:05,862 WARN [M:1;jenkins-hbase4:40781] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:05,862 INFO [M:1;jenkins-hbase4:40781] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:05,862 INFO [M:1;jenkins-hbase4:40781] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:04:05,862 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:05,863 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:05,863 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:05,863 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:05,863 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:05,863 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=23.35 KB heapSize=28.81 KB 2023-07-12 19:04:05,957 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:05,957 INFO [RS:0;jenkins-hbase4:35493] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,35493,1689188637043; zookeeper connection closed. 2023-07-12 19:04:05,957 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): regionserver:35493-0x1015aeb945e0002, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:05,958 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3c51334d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3c51334d 2023-07-12 19:04:05,958 INFO [Listener at localhost/35631] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 19:04:06,276 INFO [M:1;jenkins-hbase4:40781] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=23.35 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7e9decd58da846099de608f5826a61ac 2023-07-12 19:04:06,285 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7e9decd58da846099de608f5826a61ac as hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7e9decd58da846099de608f5826a61ac 2023-07-12 19:04:06,292 INFO [M:1;jenkins-hbase4:40781] regionserver.HStore(1080): Added hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7e9decd58da846099de608f5826a61ac, entries=9, sequenceid=190, filesize=5.4 K 2023-07-12 19:04:06,293 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegion(2948): Finished flush of dataSize ~23.35 KB/23913, heapSize ~28.80 KB/29488, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 430ms, sequenceid=190, compaction requested=false 2023-07-12 19:04:06,295 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:06,295 DEBUG [M:1;jenkins-hbase4:40781] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:06,299 INFO [M:1;jenkins-hbase4:40781] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:06,299 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:06,300 INFO [M:1;jenkins-hbase4:40781] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40781 2023-07-12 19:04:06,302 DEBUG [M:1;jenkins-hbase4:40781] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40781,1689188636986 already deleted, retry=false 2023-07-12 19:04:06,405 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:06,405 INFO [M:1;jenkins-hbase4:40781] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,40781,1689188636986; zookeeper connection closed. 2023-07-12 19:04:06,406 DEBUG [Listener at localhost/35631-EventThread] zookeeper.ZKWatcher(600): master:40781-0x1015aeb945e0001, quorum=127.0.0.1:52458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:06,407 WARN [Listener at localhost/35631] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:04:06,411 INFO [Listener at localhost/35631] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:06,514 WARN [BP-536168004-172.31.14.131-1689188632762 heartbeating to localhost/127.0.0.1:41637] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:04:06,514 WARN [BP-536168004-172.31.14.131-1689188632762 heartbeating to localhost/127.0.0.1:41637] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-536168004-172.31.14.131-1689188632762 (Datanode Uuid 67507a56-8e55-4294-9a03-1561e177d4f6) service to localhost/127.0.0.1:41637 2023-07-12 19:04:06,516 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b/dfs/data/data1/current/BP-536168004-172.31.14.131-1689188632762] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:06,516 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/cluster_4125ac00-7f17-112d-0e9a-e2ad1bd0bf2b/dfs/data/data2/current/BP-536168004-172.31.14.131-1689188632762] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:06,552 INFO [Listener at localhost/35631] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:06,663 INFO [Listener at localhost/35631] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:04:06,685 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 19:04:06,695 INFO [Listener at localhost/35631] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDisableTableWithFailover Thread=61 (was 10) Potentially hanging thread: region-location-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/35631 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:41637 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:327) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:41637 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@40e827f6 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:41637 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:41637 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:41637 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) - Thread LEAK? -, OpenFileDescriptor=394 (was 263) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=279 (was 331), ProcessCount=170 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=8629 (was 9391) 2023-07-12 19:04:06,704 INFO [Listener at localhost/35631] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=61, OpenFileDescriptor=394, MaxFileDescriptor=60000, SystemLoadAverage=279, ProcessCount=170, AvailableMemoryMB=8628 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/hadoop.log.dir so I do NOT create it in target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/abda1228-7178-d5f1-166e-9c7988ee29a6/hadoop.tmp.dir so I do NOT create it in target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840, deleteOnExit=true 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/test.cache.data in system properties and HBase conf 2023-07-12 19:04:06,705 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:04:06,706 DEBUG [Listener at localhost/35631] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:06,706 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:04:06,707 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:04:06,708 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:06,708 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:04:06,708 INFO [Listener at localhost/35631] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:04:06,712 WARN [Listener at localhost/35631] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:06,713 WARN [Listener at localhost/35631] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:06,761 WARN [Listener at localhost/35631] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:06,764 INFO [Listener at localhost/35631] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:06,772 INFO [Listener at localhost/35631] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/java.io.tmpdir/Jetty_localhost_34813_hdfs____beasxv/webapp 2023-07-12 19:04:06,893 INFO [Listener at localhost/35631] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34813 2023-07-12 19:04:06,898 WARN [Listener at localhost/35631] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:06,898 WARN [Listener at localhost/35631] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:06,951 WARN [Listener at localhost/33037] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:06,971 WARN [Listener at localhost/33037] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:04:06,973 WARN [Listener at localhost/33037] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:06,975 INFO [Listener at localhost/33037] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:06,982 INFO [Listener at localhost/33037] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/java.io.tmpdir/Jetty_localhost_33443_datanode____h5m3sq/webapp 2023-07-12 19:04:07,075 INFO [Listener at localhost/33037] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33443 2023-07-12 19:04:07,081 WARN [Listener at localhost/33639] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:07,198 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x21773632425213a4: Processing first storage report for DS-7b696925-c066-4aa9-abda-92eea1ef5a33 from datanode 2942d0b1-535c-4c80-9cfe-755dac29d448 2023-07-12 19:04:07,198 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x21773632425213a4: from storage DS-7b696925-c066-4aa9-abda-92eea1ef5a33 node DatanodeRegistration(127.0.0.1:32973, datanodeUuid=2942d0b1-535c-4c80-9cfe-755dac29d448, infoPort=41889, infoSecurePort=0, ipcPort=33639, storageInfo=lv=-57;cid=testClusterID;nsid=1214426509;c=1689188646715), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:07,198 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x21773632425213a4: Processing first storage report for DS-aa3161e6-3277-4e17-b325-c3a34fc6334e from datanode 2942d0b1-535c-4c80-9cfe-755dac29d448 2023-07-12 19:04:07,199 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x21773632425213a4: from storage DS-aa3161e6-3277-4e17-b325-c3a34fc6334e node DatanodeRegistration(127.0.0.1:32973, datanodeUuid=2942d0b1-535c-4c80-9cfe-755dac29d448, infoPort=41889, infoSecurePort=0, ipcPort=33639, storageInfo=lv=-57;cid=testClusterID;nsid=1214426509;c=1689188646715), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:07,291 DEBUG [Listener at localhost/33639] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d 2023-07-12 19:04:07,305 INFO [Listener at localhost/33639] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840/zookeeper_0, clientPort=50958, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:04:07,307 INFO [Listener at localhost/33639] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50958 2023-07-12 19:04:07,308 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,309 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,751 INFO [Listener at localhost/33639] util.FSUtils(471): Created version file at hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87 with version=8 2023-07-12 19:04:07,751 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase-staging 2023-07-12 19:04:07,753 INFO [Listener at localhost/33639] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:07,753 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,753 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,753 INFO [Listener at localhost/33639] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:07,753 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,754 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:07,754 INFO [Listener at localhost/33639] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:07,755 INFO [Listener at localhost/33639] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46733 2023-07-12 19:04:07,755 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,757 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,758 INFO [Listener at localhost/33639] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46733 connecting to ZooKeeper ensemble=127.0.0.1:50958 2023-07-12 19:04:07,765 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:467330x0, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:07,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46733-0x1015aebc48a0000 connected 2023-07-12 19:04:07,779 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:07,780 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:07,780 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:07,787 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46733 2023-07-12 19:04:07,787 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46733 2023-07-12 19:04:07,787 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46733 2023-07-12 19:04:07,788 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46733 2023-07-12 19:04:07,790 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46733 2023-07-12 19:04:07,790 INFO [Listener at localhost/33639] master.HMaster(444): hbase.rootdir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87, hbase.cluster.distributed=false 2023-07-12 19:04:07,793 INFO [Listener at localhost/33639] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:07,793 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,793 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,793 INFO [Listener at localhost/33639] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:07,794 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,794 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:07,794 INFO [Listener at localhost/33639] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:07,799 INFO [Listener at localhost/33639] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40863 2023-07-12 19:04:07,799 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,801 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,802 INFO [Listener at localhost/33639] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40863 connecting to ZooKeeper ensemble=127.0.0.1:50958 2023-07-12 19:04:07,807 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:408630x0, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:07,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40863-0x1015aebc48a0001 connected 2023-07-12 19:04:07,810 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:07,811 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:07,811 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:07,812 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40863 2023-07-12 19:04:07,812 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40863 2023-07-12 19:04:07,812 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40863 2023-07-12 19:04:07,812 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40863 2023-07-12 19:04:07,813 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40863 2023-07-12 19:04:07,813 INFO [Listener at localhost/33639] master.HMaster(444): hbase.rootdir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87, hbase.cluster.distributed=false 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:07,827 INFO [Listener at localhost/33639] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:07,828 INFO [Listener at localhost/33639] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:07,829 INFO [Listener at localhost/33639] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34749 2023-07-12 19:04:07,829 INFO [Listener at localhost/33639] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:04:07,830 DEBUG [Listener at localhost/33639] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:04:07,831 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,832 INFO [Listener at localhost/33639] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:07,833 INFO [Listener at localhost/33639] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34749 connecting to ZooKeeper ensemble=127.0.0.1:50958 2023-07-12 19:04:07,836 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:347490x0, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:07,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34749-0x1015aebc48a0002 connected 2023-07-12 19:04:07,837 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:07,837 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:07,838 DEBUG [Listener at localhost/33639] zookeeper.ZKUtil(164): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:07,838 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34749 2023-07-12 19:04:07,839 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34749 2023-07-12 19:04:07,840 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34749 2023-07-12 19:04:07,840 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34749 2023-07-12 19:04:07,841 DEBUG [Listener at localhost/33639] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34749 2023-07-12 19:04:07,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:07,842 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:07,844 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:07,844 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:07,845 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:07,846 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:07,847 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:07,847 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:07,847 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:07,847 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:07,847 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:07,847 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:07,848 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:07,848 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46733,1689188647752 from backup master directory 2023-07-12 19:04:07,850 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:07,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:07,850 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:07,850 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:07,851 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:07,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:07,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:07,858 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase4.apache.org,46733,1689188647752; waiting to become the next active master 2023-07-12 19:04:08,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/hbase.id with ID: 2c231c37-830f-4e79-9e67-64fd0d3460e2 2023-07-12 19:04:08,300 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:08,303 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:08,303 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:08,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x75118711 to 127.0.0.1:50958 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:08,717 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@785427e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:08,717 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:08,718 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:08,718 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:08,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store-tmp 2023-07-12 19:04:09,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:09,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:09,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:09,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:09,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:09,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:09,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:09,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:09,131 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:09,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46733%2C1689188647752, suffix=, logDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752, archiveDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:09,153 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK] 2023-07-12 19:04:09,157 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:09,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK]] 2023-07-12 19:04:09,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:09,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:09,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,160 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,162 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:09,163 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:09,164 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:09,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,170 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:09,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:09,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64501180, jitterRate=-0.03885751962661743}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:09,174 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:09,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:09,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:09,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:09,176 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:09,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:09,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 19:04:09,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:09,184 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:04:09,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:09,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:09,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:09,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:09,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:09,198 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:09,200 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,200 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:09,201 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:09,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:09,203 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:09,203 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:09,203 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,203 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:09,203 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,204 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46733,1689188647752, sessionid=0x1015aebc48a0000, setting cluster-up flag (Was=false) 2023-07-12 19:04:09,210 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegionServer(951): ClusterId : 2c231c37-830f-4e79-9e67-64fd0d3460e2 2023-07-12 19:04:09,212 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,212 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:09,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:09,221 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,221 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:09,226 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:09,227 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:09,228 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.hbase-snapshot/.tmp 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:09,230 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188679231 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:09,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:09,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:09,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:09,232 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:09,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:09,233 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:04:09,234 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:09,234 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:09,235 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:09,235 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188649235,5,FailOnTimeoutGroup] 2023-07-12 19:04:09,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188649235,5,FailOnTimeoutGroup] 2023-07-12 19:04:09,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,238 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:09,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,239 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,244 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(951): ClusterId : 2c231c37-830f-4e79-9e67-64fd0d3460e2 2023-07-12 19:04:09,245 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:04:09,247 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:04:09,247 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:04:09,249 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:04:09,250 DEBUG [RS:0;jenkins-hbase4:34749] zookeeper.ReadOnlyZKClient(139): Connect 0x49da4b57 to 127.0.0.1:50958 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:09,254 DEBUG [RS:0;jenkins-hbase4:34749] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43222027, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:09,254 DEBUG [RS:0;jenkins-hbase4:34749] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@620b627, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:09,263 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34749 2023-07-12 19:04:09,263 INFO [RS:0;jenkins-hbase4:34749] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:04:09,264 INFO [RS:0;jenkins-hbase4:34749] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:04:09,264 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:04:09,264 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46733,1689188647752 with isa=jenkins-hbase4.apache.org/172.31.14.131:34749, startcode=1689188647826 2023-07-12 19:04:09,264 DEBUG [RS:0;jenkins-hbase4:34749] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:09,267 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54195, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:09,267 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46733] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,268 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87 2023-07-12 19:04:09,268 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33037 2023-07-12 19:04:09,268 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:04:09,270 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:09,270 DEBUG [RS:0;jenkins-hbase4:34749] zookeeper.ZKUtil(162): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,270 WARN [RS:0;jenkins-hbase4:34749] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:09,271 INFO [RS:0;jenkins-hbase4:34749] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:09,271 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/WALs/jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,271 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34749,1689188647826] 2023-07-12 19:04:09,274 DEBUG [RS:0;jenkins-hbase4:34749] zookeeper.ZKUtil(162): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,275 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:04:09,276 INFO [RS:0;jenkins-hbase4:34749] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:04:09,277 INFO [RS:0;jenkins-hbase4:34749] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:04:09,278 INFO [RS:0;jenkins-hbase4:34749] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:04:09,278 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,278 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:04:09,279 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,279 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,279 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,279 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,279 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 DEBUG [RS:0;jenkins-hbase4:34749] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:09,280 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,280 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,280 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,292 INFO [RS:0;jenkins-hbase4:34749] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:04:09,292 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34749,1689188647826-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:09,304 INFO [RS:0;jenkins-hbase4:34749] regionserver.Replication(203): jenkins-hbase4.apache.org,34749,1689188647826 started 2023-07-12 19:04:09,304 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,34749,1689188647826, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34749, sessionid=0x1015aebc48a0002 2023-07-12 19:04:09,304 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:04:09,304 DEBUG [RS:0;jenkins-hbase4:34749] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,304 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34749,1689188647826' 2023-07-12 19:04:09,304 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:04:09,304 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34749,1689188647826' 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:04:09,305 DEBUG [RS:0;jenkins-hbase4:34749] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:04:09,306 DEBUG [RS:0;jenkins-hbase4:34749] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:04:09,306 INFO [RS:0;jenkins-hbase4:34749] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 19:04:09,306 INFO [RS:0;jenkins-hbase4:34749] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 19:04:09,409 INFO [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34749%2C1689188647826, suffix=, logDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/WALs/jenkins-hbase4.apache.org,34749,1689188647826, archiveDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/oldWALs, maxLogs=32 2023-07-12 19:04:09,433 DEBUG [RS-EventLoopGroup-8-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK] 2023-07-12 19:04:09,435 INFO [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/WALs/jenkins-hbase4.apache.org,34749,1689188647826/jenkins-hbase4.apache.org%2C34749%2C1689188647826.1689188649411 2023-07-12 19:04:09,436 DEBUG [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK]] 2023-07-12 19:04:09,648 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:09,648 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:09,649 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87 2023-07-12 19:04:10,059 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:10,060 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:10,063 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/info 2023-07-12 19:04:10,063 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:10,064 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,064 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:10,065 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:10,066 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:10,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,066 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:10,068 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/table 2023-07-12 19:04:10,068 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:10,069 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,070 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740 2023-07-12 19:04:10,071 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740 2023-07-12 19:04:10,073 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:10,075 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:10,077 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:10,078 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62676693, jitterRate=-0.06604449450969696}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:10,078 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:10,078 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:10,078 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:10,080 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:10,080 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:04:10,080 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:04:10,081 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:04:10,082 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:04:10,233 DEBUG [jenkins-hbase4:46733] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:04:10,233 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34749,1689188647826, state=OPENING 2023-07-12 19:04:10,235 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:04:10,236 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:10,236 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:10,237 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:10,237 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:10,237 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34749,1689188647826}] 2023-07-12 19:04:10,390 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:10,391 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:10,394 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51074, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:10,398 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:04:10,398 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:10,400 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34749%2C1689188647826.meta, suffix=.meta, logDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/WALs/jenkins-hbase4.apache.org,34749,1689188647826, archiveDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/oldWALs, maxLogs=32 2023-07-12 19:04:10,417 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK] 2023-07-12 19:04:10,419 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/WALs/jenkins-hbase4.apache.org,34749,1689188647826/jenkins-hbase4.apache.org%2C34749%2C1689188647826.meta.1689188650402.meta 2023-07-12 19:04:10,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK]] 2023-07-12 19:04:10,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:10,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:04:10,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:04:10,420 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:04:10,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:04:10,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:10,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:04:10,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:04:10,422 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:10,424 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/info 2023-07-12 19:04:10,424 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/info 2023-07-12 19:04:10,424 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:10,425 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,425 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:10,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:10,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:10,426 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:10,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,427 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:10,428 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/table 2023-07-12 19:04:10,428 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/table 2023-07-12 19:04:10,429 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:10,429 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:10,430 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740 2023-07-12 19:04:10,433 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740 2023-07-12 19:04:10,435 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:10,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:10,438 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61280822, jitterRate=-0.08684459328651428}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:10,438 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:10,439 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689188650390 2023-07-12 19:04:10,443 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:04:10,443 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:04:10,444 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,34749,1689188647826, state=OPEN 2023-07-12 19:04:10,453 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:10,454 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:10,454 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:10,454 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:10,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:04:10,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,34749,1689188647826 in 217 msec 2023-07-12 19:04:10,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:04:10,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 377 msec 2023-07-12 19:04:10,462 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2320 sec 2023-07-12 19:04:10,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188650462, completionTime=-1 2023-07-12 19:04:10,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:10,463 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:10,465 DEBUG [hconnection-0x6ac9dad4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:10,468 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51084, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:10,469 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:10,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188710470 2023-07-12 19:04:10,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188770470 2023-07-12 19:04:10,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-07-12 19:04:10,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46733,1689188647752-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46733,1689188647752-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46733,1689188647752-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46733, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:04:10,476 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:10,477 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:10,477 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:10,479 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:10,480 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:10,482 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:10,483 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea empty. 2023-07-12 19:04:10,483 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:10,483 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:04:10,848 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:10,895 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:10,897 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => fe0fb0d3328c6179178348543a1025ea, NAME => 'hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing fe0fb0d3328c6179178348543a1025ea, disabling compactions & flushes 2023-07-12 19:04:10,905 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. after waiting 0 ms 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:10,905 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:10,905 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for fe0fb0d3328c6179178348543a1025ea: 2023-07-12 19:04:10,908 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:10,909 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188650909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188650909"}]},"ts":"1689188650909"} 2023-07-12 19:04:10,911 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:10,912 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:10,913 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188650912"}]},"ts":"1689188650912"} 2023-07-12 19:04:10,914 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:04:10,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fe0fb0d3328c6179178348543a1025ea, ASSIGN}] 2023-07-12 19:04:10,921 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fe0fb0d3328c6179178348543a1025ea, ASSIGN 2023-07-12 19:04:10,921 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=fe0fb0d3328c6179178348543a1025ea, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34749,1689188647826; forceNewPlan=false, retain=false 2023-07-12 19:04:11,072 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fe0fb0d3328c6179178348543a1025ea, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:11,073 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188651072"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188651072"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188651072"}]},"ts":"1689188651072"} 2023-07-12 19:04:11,075 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure fe0fb0d3328c6179178348543a1025ea, server=jenkins-hbase4.apache.org,34749,1689188647826}] 2023-07-12 19:04:11,232 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:11,232 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fe0fb0d3328c6179178348543a1025ea, NAME => 'hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:11,232 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,232 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:11,233 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,233 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,234 INFO [StoreOpener-fe0fb0d3328c6179178348543a1025ea-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,236 DEBUG [StoreOpener-fe0fb0d3328c6179178348543a1025ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/info 2023-07-12 19:04:11,236 DEBUG [StoreOpener-fe0fb0d3328c6179178348543a1025ea-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/info 2023-07-12 19:04:11,236 INFO [StoreOpener-fe0fb0d3328c6179178348543a1025ea-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fe0fb0d3328c6179178348543a1025ea columnFamilyName info 2023-07-12 19:04:11,237 INFO [StoreOpener-fe0fb0d3328c6179178348543a1025ea-1] regionserver.HStore(310): Store=fe0fb0d3328c6179178348543a1025ea/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:11,238 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,239 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,241 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:11,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:11,244 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fe0fb0d3328c6179178348543a1025ea; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62498802, jitterRate=-0.06869527697563171}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:11,244 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fe0fb0d3328c6179178348543a1025ea: 2023-07-12 19:04:11,245 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea., pid=6, masterSystemTime=1689188651227 2023-07-12 19:04:11,248 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:11,248 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:11,248 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fe0fb0d3328c6179178348543a1025ea, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:11,249 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188651248"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188651248"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188651248"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188651248"}]},"ts":"1689188651248"} 2023-07-12 19:04:11,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:04:11,252 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure fe0fb0d3328c6179178348543a1025ea, server=jenkins-hbase4.apache.org,34749,1689188647826 in 175 msec 2023-07-12 19:04:11,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:04:11,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=fe0fb0d3328c6179178348543a1025ea, ASSIGN in 334 msec 2023-07-12 19:04:11,256 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:11,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188651256"}]},"ts":"1689188651256"} 2023-07-12 19:04:11,258 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:04:11,260 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:11,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 784 msec 2023-07-12 19:04:11,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:04:11,280 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:11,280 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:11,280 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:11,284 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:11,292 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:11,297 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 13 msec 2023-07-12 19:04:11,307 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:11,314 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:11,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 10 msec 2023-07-12 19:04:11,322 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:11,323 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.472sec 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46733,1689188647752-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:11,324 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46733,1689188647752-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:11,326 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:11,345 DEBUG [Listener at localhost/33639] zookeeper.ReadOnlyZKClient(139): Connect 0x10bb089a to 127.0.0.1:50958 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:11,349 DEBUG [Listener at localhost/33639] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57372864, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:11,351 DEBUG [hconnection-0x65a9954-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:11,353 INFO [RS-EventLoopGroup-8-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51088, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:11,355 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:11,355 WARN [Listener at localhost/33639] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:11,355 WARN [Listener at localhost/33639] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 19:04:11,357 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,358 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:11,358 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:11,358 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,358 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:11,358 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:11,607 INFO [Listener at localhost/33639] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,607 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:11,607 INFO [Listener at localhost/33639] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:11,607 INFO [Listener at localhost/33639] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:11,608 ERROR [Listener at localhost/33639] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-07-12 19:04:11,608 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:11,608 INFO [Listener at localhost/33639] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:11,609 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:11,609 INFO [Listener at localhost/33639] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:11,609 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:11,610 DEBUG [Listener at localhost/33639] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[] 2023-07-12 19:04:11,611 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:11,611 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:11,611 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:11,611 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:11,612 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,612 DEBUG [Listener at localhost/33639] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:11,612 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:11,613 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:11,613 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,34749,1689188647826, table=hbase:meta, region=1588230740 2023-07-12 19:04:11,613 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188681613 2023-07-12 19:04:11,614 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:11,614 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,617 DEBUG [Listener at localhost/33639] assignment.RegionStateStore(147): Load hbase:meta entry region=fe0fb0d3328c6179178348543a1025ea, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34749,1689188647826, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826, openSeqNum=2 2023-07-12 19:04:11,617 INFO [Listener at localhost/33639] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:11,617 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188711617 2023-07-12 19:04:11,617 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188771617 2023-07-12 19:04:11,617 INFO [Listener at localhost/33639] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:11,617 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:11,617 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:11,619 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:11,621 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:11,622 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 empty. 2023-07-12 19:04:11,623 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:11,623 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-12 19:04:11,641 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:11,642 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4da06f8e4b1d50ea84fa72b4dd1ab920, NAME => 'testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 4da06f8e4b1d50ea84fa72b4dd1ab920, disabling compactions & flushes 2023-07-12 19:04:11,671 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. after waiting 0 ms 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:11,671 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:11,671 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 4da06f8e4b1d50ea84fa72b4dd1ab920: 2023-07-12 19:04:11,674 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:11,674 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:11,674 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:11,867 INFO [Listener at localhost/33639] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,867 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:11,867 INFO [Listener at localhost/33639] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:11,867 INFO [Listener at localhost/33639] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:11,868 ERROR [Listener at localhost/33639] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] 2023-07-12 19:04:11,868 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:11,868 INFO [Listener at localhost/33639] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:11,868 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:11,868 INFO [Listener at localhost/33639] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:11,869 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:11,870 DEBUG [Listener at localhost/33639] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0] 2023-07-12 19:04:11,870 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:11,870 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:11,871 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:11,871 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:11,871 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,871 DEBUG [Listener at localhost/33639] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:11,871 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:11,872 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:11,873 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,34749,1689188647826, table=hbase:meta, region=1588230740 2023-07-12 19:04:11,873 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188681873 2023-07-12 19:04:11,874 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:11,874 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:11,876 DEBUG [Listener at localhost/33639] assignment.RegionStateStore(147): Load hbase:meta entry region=fe0fb0d3328c6179178348543a1025ea, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34749,1689188647826, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826, openSeqNum=2 2023-07-12 19:04:11,876 INFO [Listener at localhost/33639] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:11,876 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188711876 2023-07-12 19:04:11,876 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188771876 2023-07-12 19:04:11,876 INFO [Listener at localhost/33639] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 19:04:11,877 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:11,881 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:11,882 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 empty. 2023-07-12 19:04:11,883 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:11,883 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCreateWithFailoverAtStep3 regions 2023-07-12 19:04:12,295 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp/data/default/testCreateWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:12,297 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4da06f8e4b1d50ea84fa72b4dd1ab920, NAME => 'testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testCreateWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.tmp 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 4da06f8e4b1d50ea84fa72b4dd1ab920, disabling compactions & flushes 2023-07-12 19:04:12,706 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. after waiting 0 ms 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:12,706 INFO [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:12,706 DEBUG [RegionOpenAndInit-testCreateWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 4da06f8e4b1d50ea84fa72b4dd1ab920: 2023-07-12 19:04:12,708 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:12,710 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:12,712 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188652711"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188652711"}]},"ts":"1689188652711"} 2023-07-12 19:04:12,714 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:12,714 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:12,714 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:12,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:12,714 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:12,877 ERROR [Listener at localhost/33639] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:12,877 INFO [Listener at localhost/33639] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:12,878 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:12,878 INFO [Listener at localhost/33639] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:12,879 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:12,879 DEBUG [Listener at localhost/33639] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1] 2023-07-12 19:04:12,880 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:12,880 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:12,880 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:12,881 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:12,881 DEBUG [Listener at localhost/33639] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:12,881 DEBUG [Listener at localhost/33639] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:12,881 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:12,882 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:12,883 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,34749,1689188647826, table=hbase:meta, region=1588230740 2023-07-12 19:04:12,884 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188682884 2023-07-12 19:04:12,884 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:12,884 DEBUG [Listener at localhost/33639] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:12,887 DEBUG [Listener at localhost/33639] assignment.RegionStateStore(147): Load hbase:meta entry region=fe0fb0d3328c6179178348543a1025ea, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34749,1689188647826, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826, openSeqNum=2 2023-07-12 19:04:12,888 DEBUG [Listener at localhost/33639] assignment.RegionStateStore(147): Load hbase:meta entry region=4da06f8e4b1d50ea84fa72b4dd1ab920, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:12,888 INFO [Listener at localhost/33639] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:12,888 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188712888 2023-07-12 19:04:12,888 INFO [Listener at localhost/33639] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188772888 2023-07-12 19:04:12,888 INFO [Listener at localhost/33639] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:12,888 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:12,889 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188652889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188652889"}]},"ts":"1689188652889"} 2023-07-12 19:04:12,891 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:12,891 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:12,893 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:12,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188652894"}]},"ts":"1689188652894"} 2023-07-12 19:04:12,895 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:12,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=4da06f8e4b1d50ea84fa72b4dd1ab920, ASSIGN}] 2023-07-12 19:04:12,899 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:12,899 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:12,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:12,899 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=9, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:13,138 INFO [Listener at localhost/33639] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 19:04:13,139 INFO [Listener at localhost/33639] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:13,139 ERROR [Listener at localhost/33639] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 19:04:13,139 ERROR [Listener at localhost/33639] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase4.apache.org,46733,1689188647752: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:121) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testCreateWithFailover(TestMasterFailoverWithProcedures.java:104) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:13,139 INFO [Listener at localhost/33639] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46733,1689188647752' ***** 2023-07-12 19:04:13,139 INFO [Listener at localhost/33639] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/33639 2023-07-12 19:04:13,140 DEBUG [M:0;jenkins-hbase4:46733] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@96d38b6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:13,140 INFO [M:0;jenkins-hbase4:46733] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,46733,1689188647752 2023-07-12 19:04:13,140 DEBUG [M:0;jenkins-hbase4:46733] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75118711 to 127.0.0.1:50958 2023-07-12 19:04:13,140 DEBUG [M:0;jenkins-hbase4:46733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:13,141 INFO [M:0;jenkins-hbase4:46733] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46733,1689188647752; all regions closed. 2023-07-12 19:04:13,141 DEBUG [M:0;jenkins-hbase4:46733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:13,141 DEBUG [M:0;jenkins-hbase4:46733] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:13,141 DEBUG [M:0;jenkins-hbase4:46733] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:13,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188649235] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188649235,5,FailOnTimeoutGroup] 2023-07-12 19:04:13,141 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:13,141 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188649235] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188649235,5,FailOnTimeoutGroup] 2023-07-12 19:04:13,141 INFO [M:0;jenkins-hbase4:46733] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:13,142 INFO [M:0;jenkins-hbase4:46733] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:13,142 INFO [M:0;jenkins-hbase4:46733] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:13,142 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,142 DEBUG [M:0;jenkins-hbase4:46733] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:13,142 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:13,142 INFO [M:0;jenkins-hbase4:46733] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:13,142 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,143 ERROR [M:0;jenkins-hbase4:46733] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:13,142 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,143 INFO [M:0;jenkins-hbase4:46733] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 19:04:13,143 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:13,143 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:13,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:13,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:13,244 DEBUG [M:0;jenkins-hbase4:46733] zookeeper.ZKUtil(398): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:13,244 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 19:04:13,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:13,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:13,244 WARN [M:0;jenkins-hbase4:46733] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:13,244 INFO [M:0;jenkins-hbase4:46733] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:13,245 INFO [M:0;jenkins-hbase4:46733] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 19:04:13,246 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,246 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,246 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:13,246 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:13,246 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:13,246 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:13,247 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40863,1689188647793 from backup master directory 2023-07-12 19:04:13,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:13,249 DEBUG [M:0;jenkins-hbase4:46733] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:13,250 INFO [M:0;jenkins-hbase4:46733] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:13,250 DEBUG [M:0;jenkins-hbase4:46733] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:13,250 DEBUG [M:0;jenkins-hbase4:46733] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:13,250 DEBUG [M:0;jenkins-hbase4:46733] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:13,250 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:13,250 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:13,250 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:13,250 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:13,250 ERROR [M:0;jenkins-hbase4:46733] regionserver.HRegion(1825): Memstore data size is 29049 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:13,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:13,250 INFO [M:0;jenkins-hbase4:46733] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:13,250 DEBUG [M:0;jenkins-hbase4:46733] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:13,250 INFO [M:0;jenkins-hbase4:46733] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:13,250 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:13,251 INFO [M:0;jenkins-hbase4:46733] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46733 2023-07-12 19:04:13,255 DEBUG [M:0;jenkins-hbase4:46733] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46733,1689188647752 already deleted, retry=false 2023-07-12 19:04:13,268 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:13,299 WARN [RS-EventLoopGroup-8-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46733 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46733 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:13,300 DEBUG [RS-EventLoopGroup-8-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46733 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46733 2023-07-12 19:04:13,357 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:13,357 INFO [M:0;jenkins-hbase4:46733] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46733,1689188647752; zookeeper connection closed. 2023-07-12 19:04:13,357 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:46733-0x1015aebc48a0000, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:13,401 DEBUG [RS:0;jenkins-hbase4:34749] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:13,403 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45825, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:13,404 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:13,506 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:13,608 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:13,688 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x195023a1 to 127.0.0.1:50958 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:13,693 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67091062, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:13,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:13,694 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:13,694 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:13,700 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752 to hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752-dead as it is dead 2023-07-12 19:04:13,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752-dead/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:13,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752-dead/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 after 1ms 2023-07-12 19:04:13,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752-dead/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 to hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:13,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,46733,1689188647752-dead 2023-07-12 19:04:13,703 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:13,705 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40863%2C1689188647793, suffix=, logDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,40863,1689188647793, archiveDir=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:13,711 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:13,719 DEBUG [RS-EventLoopGroup-8-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK] 2023-07-12 19:04:13,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/WALs/jenkins-hbase4.apache.org,40863,1689188647793/jenkins-hbase4.apache.org%2C40863%2C1689188647793.1689188653705 2023-07-12 19:04:13,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32973,DS-7b696925-c066-4aa9-abda-92eea1ef5a33,DISK]] 2023-07-12 19:04:13,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:13,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:13,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:13,724 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:13,727 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:13,728 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:13,729 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:13,729 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:13,730 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 19:04:13,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:13,737 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 76, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=70, path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:13,737 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=28.37 KB heapSize=34.23 KB 2023-07-12 19:04:13,813 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:13,916 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:14,019 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:14,122 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40863] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:14,150 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=28.37 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/06052dfb5f7c416bb2db113c08de0cc9 2023-07-12 19:04:14,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/06052dfb5f7c416bb2db113c08de0cc9 as hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/06052dfb5f7c416bb2db113c08de0cc9 2023-07-12 19:04:14,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/06052dfb5f7c416bb2db113c08de0cc9, entries=76, sequenceid=70, filesize=33.2 K 2023-07-12 19:04:14,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~28.37 KB/29049, heapSize ~34.21 KB/35032, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 427ms, sequenceid=70, compaction requested=false; wal=null 2023-07-12 19:04:14,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46733%2C1689188647752.1689188649136 2023-07-12 19:04:14,168 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:14,170 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/70.seqid, newMaxSeqId=70, maxSeqId=1 2023-07-12 19:04:14,171 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=71; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64331250, jitterRate=-0.041389673948287964}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:14,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:14,172 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:14,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:14,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:14,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:14,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:14,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=9, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.CreateTableProcedure) stack ids=[0, 1, 2] 2023-07-12 19:04:14,177 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:14,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:14,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:14,178 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:14,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:14,179 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.Procedure(966): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:14,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-07-12 19:04:14,179 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:14,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:14,180 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,34749,1689188647826, table=hbase:meta, region=1588230740 2023-07-12 19:04:14,181 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:14,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:14,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:14,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:14,197 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:14,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:14,198 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:14,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:14,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:14,200 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:14,200 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40863,1689188647793, sessionid=0x1015aebc48a0001, setting cluster-up flag (Was=true) 2023-07-12 19:04:14,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:14,202 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:14,204 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:14,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:14,206 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/.hbase-snapshot/.tmp 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:14,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:14,209 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188684209 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:14,211 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=testCreateWithFailoverAtStep3 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:14,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188654212,5,FailOnTimeoutGroup] 2023-07-12 19:04:14,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188654212,5,FailOnTimeoutGroup] 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188654212, completionTime=-1 2023-07-12 19:04:14,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:14,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:14,214 DEBUG [hconnection-0x1cca3a09-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:14,216 INFO [RS-EventLoopGroup-8-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51092, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:14,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=fe0fb0d3328c6179178348543a1025ea, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34749,1689188647826, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826, openSeqNum=2 2023-07-12 19:04:14,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=4da06f8e4b1d50ea84fa72b4dd1ab920, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:14,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:14,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188714218 2023-07-12 19:04:14,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188774218 2023-07-12 19:04:14,218 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 19:04:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1689188647793-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1689188647793-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1689188647793-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40863, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:14,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 19:04:14,234 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:14,240 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:14,244 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:14,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.994sec 2023-07-12 19:04:14,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:14,244 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:14,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:14,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:14,244 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1689188647793-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:14,244 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188654244"}]},"ts":"1689188654244"} 2023-07-12 19:04:14,245 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40863,1689188647793-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:14,247 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:14,247 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:14,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=4da06f8e4b1d50ea84fa72b4dd1ab920, ASSIGN}] 2023-07-12 19:04:14,251 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=4da06f8e4b1d50ea84fa72b4dd1ab920, ASSIGN 2023-07-12 19:04:14,252 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=4da06f8e4b1d50ea84fa72b4dd1ab920, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34749,1689188647826; forceNewPlan=false, retain=false 2023-07-12 19:04:14,403 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4da06f8e4b1d50ea84fa72b4dd1ab920, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:14,403 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188654403"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188654403"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188654403"}]},"ts":"1689188654403"} 2023-07-12 19:04:14,406 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 4da06f8e4b1d50ea84fa72b4dd1ab920, server=jenkins-hbase4.apache.org,34749,1689188647826}] 2023-07-12 19:04:14,559 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:14,560 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:14,562 INFO [RS-EventLoopGroup-8-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:14,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:14,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4da06f8e4b1d50ea84fa72b4dd1ab920, NAME => 'testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:14,566 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCreateWithFailoverAtStep3 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:14,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,568 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,569 DEBUG [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/f1 2023-07-12 19:04:14,569 DEBUG [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/f1 2023-07-12 19:04:14,570 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4da06f8e4b1d50ea84fa72b4dd1ab920 columnFamilyName f1 2023-07-12 19:04:14,570 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] regionserver.HStore(310): Store=4da06f8e4b1d50ea84fa72b4dd1ab920/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:14,570 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,572 DEBUG [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/f2 2023-07-12 19:04:14,572 DEBUG [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/f2 2023-07-12 19:04:14,572 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4da06f8e4b1d50ea84fa72b4dd1ab920 columnFamilyName f2 2023-07-12 19:04:14,573 INFO [StoreOpener-4da06f8e4b1d50ea84fa72b4dd1ab920-1] regionserver.HStore(310): Store=4da06f8e4b1d50ea84fa72b4dd1ab920/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:14,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,574 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testCreateWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:14,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:14,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4da06f8e4b1d50ea84fa72b4dd1ab920; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74169073, jitterRate=0.10520531237125397}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:14,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4da06f8e4b1d50ea84fa72b4dd1ab920: 2023-07-12 19:04:14,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920., pid=11, masterSystemTime=1689188654559 2023-07-12 19:04:14,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:14,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:14,584 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=4da06f8e4b1d50ea84fa72b4dd1ab920, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:14,585 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188654584"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188654584"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188654584"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188654584"}]},"ts":"1689188654584"} 2023-07-12 19:04:14,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 19:04:14,589 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 4da06f8e4b1d50ea84fa72b4dd1ab920, server=jenkins-hbase4.apache.org,34749,1689188647826 in 180 msec 2023-07-12 19:04:14,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 19:04:14,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testCreateWithFailoverAtStep3, region=4da06f8e4b1d50ea84fa72b4dd1ab920, ASSIGN in 341 msec 2023-07-12 19:04:14,592 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:14,592 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCreateWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188654592"}]},"ts":"1689188654592"} 2023-07-12 19:04:14,593 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCreateWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 19:04:14,596 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCreateWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:14,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testCreateWithFailoverAtStep3 in 3.2410 sec 2023-07-12 19:04:14,644 DEBUG [Listener at localhost/33639] util.CommonFSUtils(682): File system contents for path hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3 2023-07-12 19:04:14,645 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 19:04:14,645 DEBUG [Listener at localhost/33639] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 19:04:14,646 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 19:04:14,646 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |-4da06f8e4b1d50ea84fa72b4dd1ab920/ 2023-07-12 19:04:14,647 DEBUG [Listener at localhost/33639] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:14,647 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:14,647 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:14,648 DEBUG [Listener at localhost/33639] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:14,648 DEBUG [Listener at localhost/33639] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:14,653 DEBUG [Listener at localhost/33639] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 19:04:14,658 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:04:14,658 DEBUG [Listener at localhost/33639] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10bb089a to 127.0.0.1:50958 2023-07-12 19:04:14,658 DEBUG [Listener at localhost/33639] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:14,659 DEBUG [Listener at localhost/33639] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:04:14,659 DEBUG [Listener at localhost/33639] util.JVMClusterUtil(257): Found active master hash=1930430908, stopped=false 2023-07-12 19:04:14,659 INFO [Listener at localhost/33639] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:14,662 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:14,662 INFO [Listener at localhost/33639] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:14,662 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:14,662 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:14,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:14,664 DEBUG [Listener at localhost/33639] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x195023a1 to 127.0.0.1:50958 2023-07-12 19:04:14,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:14,664 DEBUG [Listener at localhost/33639] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:14,664 INFO [Listener at localhost/33639] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34749,1689188647826' ***** 2023-07-12 19:04:14,665 INFO [Listener at localhost/33639] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:04:14,665 INFO [RS:0;jenkins-hbase4:34749] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:04:14,665 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:04:14,665 INFO [RS:0;jenkins-hbase4:34749] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:04:14,665 INFO [RS:0;jenkins-hbase4:34749] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:04:14,665 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(3305): Received CLOSE for fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(3305): Received CLOSE for 4da06f8e4b1d50ea84fa72b4dd1ab920 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:14,666 DEBUG [RS:0;jenkins-hbase4:34749] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49da4b57 to 127.0.0.1:50958 2023-07-12 19:04:14,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fe0fb0d3328c6179178348543a1025ea, disabling compactions & flushes 2023-07-12 19:04:14,666 DEBUG [RS:0;jenkins-hbase4:34749] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:14,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:04:14,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:14,666 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:04:14,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. after waiting 0 ms 2023-07-12 19:04:14,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:14,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fe0fb0d3328c6179178348543a1025ea 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 19:04:14,667 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 19:04:14,667 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1478): Online Regions={fe0fb0d3328c6179178348543a1025ea=hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea., 1588230740=hbase:meta,,1.1588230740, 4da06f8e4b1d50ea84fa72b4dd1ab920=testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920.} 2023-07-12 19:04:14,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:14,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:14,667 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1504): Waiting on 1588230740, 4da06f8e4b1d50ea84fa72b4dd1ab920, fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:14,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:14,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:14,667 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:14,667 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.09 KB heapSize=5.82 KB 2023-07-12 19:04:14,684 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:14,867 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1504): Waiting on 1588230740, 4da06f8e4b1d50ea84fa72b4dd1ab920, fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:15,068 DEBUG [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1504): Waiting on 1588230740, 4da06f8e4b1d50ea84fa72b4dd1ab920, fe0fb0d3328c6179178348543a1025ea 2023-07-12 19:04:15,085 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.82 KB at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/.tmp/info/f45d93c95bdc45eab56f345c543b36ec 2023-07-12 19:04:15,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/.tmp/info/692be775940e4358842bacaa6ae3610c 2023-07-12 19:04:15,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/.tmp/info/692be775940e4358842bacaa6ae3610c as hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/info/692be775940e4358842bacaa6ae3610c 2023-07-12 19:04:15,102 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=277 B at sequenceid=16 (bloomFilter=false), to=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/.tmp/table/57c61f2b1196435381f0660f2754fb84 2023-07-12 19:04:15,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/info/692be775940e4358842bacaa6ae3610c, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 19:04:15,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for fe0fb0d3328c6179178348543a1025ea in 437ms, sequenceid=6, compaction requested=false 2023-07-12 19:04:15,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 19:04:15,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/namespace/fe0fb0d3328c6179178348543a1025ea/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 19:04:15,110 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/.tmp/info/f45d93c95bdc45eab56f345c543b36ec as hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/info/f45d93c95bdc45eab56f345c543b36ec 2023-07-12 19:04:15,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fe0fb0d3328c6179178348543a1025ea: 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689188650476.fe0fb0d3328c6179178348543a1025ea. 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4da06f8e4b1d50ea84fa72b4dd1ab920, disabling compactions & flushes 2023-07-12 19:04:15,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. after waiting 0 ms 2023-07-12 19:04:15,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:15,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/default/testCreateWithFailoverAtStep3/4da06f8e4b1d50ea84fa72b4dd1ab920/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:15,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:15,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4da06f8e4b1d50ea84fa72b4dd1ab920: 2023-07-12 19:04:15,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testCreateWithFailoverAtStep3,,1689188651355.4da06f8e4b1d50ea84fa72b4dd1ab920. 2023-07-12 19:04:15,120 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/info/f45d93c95bdc45eab56f345c543b36ec, entries=20, sequenceid=16, filesize=7.2 K 2023-07-12 19:04:15,121 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/.tmp/table/57c61f2b1196435381f0660f2754fb84 as hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/table/57c61f2b1196435381f0660f2754fb84 2023-07-12 19:04:15,127 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/table/57c61f2b1196435381f0660f2754fb84, entries=5, sequenceid=16, filesize=4.9 K 2023-07-12 19:04:15,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.09 KB/3163, heapSize ~5.54 KB/5672, currentSize=0 B/0 for 1588230740 in 461ms, sequenceid=16, compaction requested=false 2023-07-12 19:04:15,128 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 19:04:15,135 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2023-07-12 19:04:15,135 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:15,136 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:15,136 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:15,136 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:15,268 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34749,1689188647826; all regions closed. 2023-07-12 19:04:15,274 DEBUG [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/oldWALs 2023-07-12 19:04:15,274 INFO [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34749%2C1689188647826.meta:.meta(num 1689188650402) 2023-07-12 19:04:15,278 DEBUG [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/oldWALs 2023-07-12 19:04:15,278 INFO [RS:0;jenkins-hbase4:34749] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34749%2C1689188647826:(num 1689188649411) 2023-07-12 19:04:15,278 DEBUG [RS:0;jenkins-hbase4:34749] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:15,278 INFO [RS:0;jenkins-hbase4:34749] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:15,278 INFO [RS:0;jenkins-hbase4:34749] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 19:04:15,278 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:15,279 INFO [RS:0;jenkins-hbase4:34749] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34749 2023-07-12 19:04:15,282 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34749,1689188647826 2023-07-12 19:04:15,282 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:15,282 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:15,284 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34749,1689188647826] 2023-07-12 19:04:15,284 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34749,1689188647826; numProcessing=1 2023-07-12 19:04:15,285 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34749,1689188647826 already deleted, retry=false 2023-07-12 19:04:15,285 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34749,1689188647826 expired; onlineServers=0 2023-07-12 19:04:15,285 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,40863,1689188647793' ***** 2023-07-12 19:04:15,285 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:04:15,286 DEBUG [M:1;jenkins-hbase4:40863] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@772f9306, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:15,286 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,40863,1689188647793 2023-07-12 19:04:15,286 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,40863,1689188647793; all regions closed. 2023-07-12 19:04:15,286 DEBUG [M:1;jenkins-hbase4:40863] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:15,286 DEBUG [M:1;jenkins-hbase4:40863] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:15,286 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:15,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188654212] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188654212,5,FailOnTimeoutGroup] 2023-07-12 19:04:15,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188654212] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188654212,5,FailOnTimeoutGroup] 2023-07-12 19:04:15,286 DEBUG [M:1;jenkins-hbase4:40863] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:15,287 INFO [M:1;jenkins-hbase4:40863] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:15,287 INFO [M:1;jenkins-hbase4:40863] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:15,287 INFO [M:1;jenkins-hbase4:40863] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:15,287 DEBUG [M:1;jenkins-hbase4:40863] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:15,287 INFO [M:1;jenkins-hbase4:40863] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:15,287 ERROR [M:1;jenkins-hbase4:40863] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:15,288 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:15,288 INFO [M:1;jenkins-hbase4:40863] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:15,288 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:15,288 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:15,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:15,288 DEBUG [M:1;jenkins-hbase4:40863] zookeeper.ZKUtil(398): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:15,288 WARN [M:1;jenkins-hbase4:40863] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:15,288 INFO [M:1;jenkins-hbase4:40863] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:15,289 INFO [M:1;jenkins-hbase4:40863] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:04:15,289 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:15,289 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:15,289 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:15,289 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:15,289 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:15,289 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=11.98 KB heapSize=14.01 KB 2023-07-12 19:04:15,384 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:15,384 INFO [RS:0;jenkins-hbase4:34749] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34749,1689188647826; zookeeper connection closed. 2023-07-12 19:04:15,384 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): regionserver:34749-0x1015aebc48a0002, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:15,385 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5f568103] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5f568103 2023-07-12 19:04:15,385 INFO [Listener at localhost/33639] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 19:04:15,700 INFO [M:1;jenkins-hbase4:40863] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.98 KB at sequenceid=92 (bloomFilter=true), to=hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/92c45ed480614017b17b5bb3f023b227 2023-07-12 19:04:15,707 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/92c45ed480614017b17b5bb3f023b227 as hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/92c45ed480614017b17b5bb3f023b227 2023-07-12 19:04:15,712 INFO [M:1;jenkins-hbase4:40863] regionserver.HStore(1080): Added hdfs://localhost:33037/user/jenkins/test-data/c4b8d541-fa54-e628-74c8-7db010643d87/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/92c45ed480614017b17b5bb3f023b227, entries=3, sequenceid=92, filesize=5.7 K 2023-07-12 19:04:15,713 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegion(2948): Finished flush of dataSize ~11.98 KB/12264, heapSize ~13.99 KB/14328, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 424ms, sequenceid=92, compaction requested=false 2023-07-12 19:04:15,715 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:15,715 DEBUG [M:1;jenkins-hbase4:40863] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:15,718 INFO [M:1;jenkins-hbase4:40863] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:15,718 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:15,718 INFO [M:1;jenkins-hbase4:40863] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40863 2023-07-12 19:04:15,722 DEBUG [M:1;jenkins-hbase4:40863] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40863,1689188647793 already deleted, retry=false 2023-07-12 19:04:15,826 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:15,826 INFO [M:1;jenkins-hbase4:40863] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,40863,1689188647793; zookeeper connection closed. 2023-07-12 19:04:15,826 DEBUG [Listener at localhost/33639-EventThread] zookeeper.ZKWatcher(600): master:40863-0x1015aebc48a0001, quorum=127.0.0.1:50958, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:15,827 WARN [Listener at localhost/33639] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:04:15,831 INFO [Listener at localhost/33639] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:15,923 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:15,952 WARN [BP-1799105172-172.31.14.131-1689188646715 heartbeating to localhost/127.0.0.1:33037] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:04:15,952 WARN [BP-1799105172-172.31.14.131-1689188646715 heartbeating to localhost/127.0.0.1:33037] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1799105172-172.31.14.131-1689188646715 (Datanode Uuid 2942d0b1-535c-4c80-9cfe-755dac29d448) service to localhost/127.0.0.1:33037 2023-07-12 19:04:15,952 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840/dfs/data/data1/current/BP-1799105172-172.31.14.131-1689188646715] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:15,953 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/cluster_16c928a0-d4cb-9055-4161-754e53588840/dfs/data/data2/current/BP-1799105172-172.31.14.131-1689188646715] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:15,966 INFO [Listener at localhost/33639] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:16,075 INFO [Listener at localhost/33639] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:04:16,086 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 19:04:16,096 INFO [Listener at localhost/33639] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testCreateWithFailover Thread=84 (was 61) Potentially hanging thread: LeaseRenewer:jenkins@localhost:33037 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/33639 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33037 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33037 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:33037 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33037 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=420 (was 394) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=244 (was 279), ProcessCount=170 (was 170), AvailableMemoryMB=8530 (was 8628) 2023-07-12 19:04:16,104 INFO [Listener at localhost/33639] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=84, OpenFileDescriptor=420, MaxFileDescriptor=60000, SystemLoadAverage=244, ProcessCount=170, AvailableMemoryMB=8530 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/hadoop.log.dir so I do NOT create it in target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/f0b37b5b-ad11-4b07-55c8-1adfefdbaa4d/hadoop.tmp.dir so I do NOT create it in target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17, deleteOnExit=true 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/test.cache.data in system properties and HBase conf 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:04:16,105 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:04:16,106 DEBUG [Listener at localhost/33639] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:04:16,106 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:16,107 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:04:16,108 INFO [Listener at localhost/33639] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:04:16,112 WARN [Listener at localhost/33639] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:16,112 WARN [Listener at localhost/33639] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:16,160 WARN [Listener at localhost/33639] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:16,163 INFO [Listener at localhost/33639] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:16,169 INFO [Listener at localhost/33639] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/java.io.tmpdir/Jetty_localhost_41153_hdfs____.hx9m5g/webapp 2023-07-12 19:04:16,261 INFO [Listener at localhost/33639] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41153 2023-07-12 19:04:16,271 WARN [Listener at localhost/33639] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:16,271 WARN [Listener at localhost/33639] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:16,329 WARN [Listener at localhost/42335] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:16,343 WARN [Listener at localhost/42335] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:04:16,346 WARN [Listener at localhost/42335] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:16,348 INFO [Listener at localhost/42335] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:16,355 INFO [Listener at localhost/42335] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/java.io.tmpdir/Jetty_localhost_43197_datanode____.pfxvjd/webapp 2023-07-12 19:04:16,459 INFO [Listener at localhost/42335] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43197 2023-07-12 19:04:16,467 WARN [Listener at localhost/46683] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:16,573 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdb7434373d982fe6: Processing first storage report for DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618 from datanode 269d8c42-ef61-4d65-8c79-3ccfe886cc5d 2023-07-12 19:04:16,573 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdb7434373d982fe6: from storage DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618 node DatanodeRegistration(127.0.0.1:39295, datanodeUuid=269d8c42-ef61-4d65-8c79-3ccfe886cc5d, infoPort=35713, infoSecurePort=0, ipcPort=46683, storageInfo=lv=-57;cid=testClusterID;nsid=1940993290;c=1689188656116), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:16,573 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xdb7434373d982fe6: Processing first storage report for DS-cc46717b-a630-4513-b4d7-6b58e91ccb10 from datanode 269d8c42-ef61-4d65-8c79-3ccfe886cc5d 2023-07-12 19:04:16,573 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xdb7434373d982fe6: from storage DS-cc46717b-a630-4513-b4d7-6b58e91ccb10 node DatanodeRegistration(127.0.0.1:39295, datanodeUuid=269d8c42-ef61-4d65-8c79-3ccfe886cc5d, infoPort=35713, infoSecurePort=0, ipcPort=46683, storageInfo=lv=-57;cid=testClusterID;nsid=1940993290;c=1689188656116), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:16,580 DEBUG [Listener at localhost/46683] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54 2023-07-12 19:04:16,584 INFO [Listener at localhost/46683] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17/zookeeper_0, clientPort=54458, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:04:16,586 INFO [Listener at localhost/46683] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54458 2023-07-12 19:04:16,587 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:16,588 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,001 INFO [Listener at localhost/46683] util.FSUtils(471): Created version file at hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868 with version=8 2023-07-12 19:04:17,001 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase-staging 2023-07-12 19:04:17,003 INFO [Listener at localhost/46683] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:17,003 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,003 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,004 INFO [Listener at localhost/46683] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:17,004 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,004 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:17,004 INFO [Listener at localhost/46683] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:17,005 INFO [Listener at localhost/46683] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46015 2023-07-12 19:04:17,005 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,007 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,008 INFO [Listener at localhost/46683] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46015 connecting to ZooKeeper ensemble=127.0.0.1:54458 2023-07-12 19:04:17,015 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:460150x0, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:17,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:46015-0x1015aebe8ca0000 connected 2023-07-12 19:04:17,027 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:17,027 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:17,028 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:17,028 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46015 2023-07-12 19:04:17,028 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46015 2023-07-12 19:04:17,029 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46015 2023-07-12 19:04:17,029 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46015 2023-07-12 19:04:17,029 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46015 2023-07-12 19:04:17,029 INFO [Listener at localhost/46683] master.HMaster(444): hbase.rootdir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868, hbase.cluster.distributed=false 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:17,031 INFO [Listener at localhost/46683] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:17,033 INFO [Listener at localhost/46683] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42421 2023-07-12 19:04:17,034 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,035 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,036 INFO [Listener at localhost/46683] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42421 connecting to ZooKeeper ensemble=127.0.0.1:54458 2023-07-12 19:04:17,041 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:424210x0, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:17,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42421-0x1015aebe8ca0001 connected 2023-07-12 19:04:17,044 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:17,044 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:17,045 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:17,045 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42421 2023-07-12 19:04:17,045 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42421 2023-07-12 19:04:17,046 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42421 2023-07-12 19:04:17,046 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42421 2023-07-12 19:04:17,046 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42421 2023-07-12 19:04:17,046 INFO [Listener at localhost/46683] master.HMaster(444): hbase.rootdir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868, hbase.cluster.distributed=false 2023-07-12 19:04:17,060 INFO [Listener at localhost/46683] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:17,061 INFO [Listener at localhost/46683] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:17,062 INFO [Listener at localhost/46683] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36517 2023-07-12 19:04:17,063 INFO [Listener at localhost/46683] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:04:17,063 DEBUG [Listener at localhost/46683] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:04:17,064 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,065 INFO [Listener at localhost/46683] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,066 INFO [Listener at localhost/46683] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36517 connecting to ZooKeeper ensemble=127.0.0.1:54458 2023-07-12 19:04:17,118 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:365170x0, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:17,119 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): regionserver:365170x0, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:17,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:36517-0x1015aebe8ca0002 connected 2023-07-12 19:04:17,119 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:17,120 DEBUG [Listener at localhost/46683] zookeeper.ZKUtil(164): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:17,120 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36517 2023-07-12 19:04:17,120 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36517 2023-07-12 19:04:17,121 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36517 2023-07-12 19:04:17,121 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36517 2023-07-12 19:04:17,121 DEBUG [Listener at localhost/46683] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36517 2023-07-12 19:04:17,124 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:17,124 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:17,125 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:17,125 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:17,127 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:17,127 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:17,128 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:17,128 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:17,128 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:17,129 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:17,129 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:17,129 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:17,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:17,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:17,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:17,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46015,1689188657003 from backup master directory 2023-07-12 19:04:17,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase4.apache.org,46015,1689188657003; waiting to become the next active master 2023-07-12 19:04:17,131 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:17,131 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:17,131 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:17,131 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:17,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:17,546 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/hbase.id with ID: b2bb5c2c-e1f7-476f-b609-451f2f518b82 2023-07-12 19:04:17,557 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:17,561 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:17,561 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:17,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x07055ccf to 127.0.0.1:54458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:17,973 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@797e6f21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:17,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:17,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:17,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:17,975 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store-tmp 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:17,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:17,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:17,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:17,982 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:17,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46015%2C1689188657003, suffix=, logDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003, archiveDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:18,002 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK] 2023-07-12 19:04:18,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:18,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK]] 2023-07-12 19:04:18,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:18,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:18,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,006 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,007 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:18,008 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:18,008 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:18,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:18,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:18,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61608189, jitterRate=-0.08196644484996796}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:18,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:18,014 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:18,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:18,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:18,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:18,016 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:18,016 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 19:04:18,016 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:18,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:04:18,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:18,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:18,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:18,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:18,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:18,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:18,031 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,031 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:18,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:18,032 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:18,034 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:18,034 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:18,034 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,034 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,034 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:18,034 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46015,1689188657003, sessionid=0x1015aebe8ca0000, setting cluster-up flag (Was=false) 2023-07-12 19:04:18,042 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegionServer(951): ClusterId : b2bb5c2c-e1f7-476f-b609-451f2f518b82 2023-07-12 19:04:18,043 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,043 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:18,048 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:18,051 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,051 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:18,055 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:18,056 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:18,057 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.hbase-snapshot/.tmp 2023-07-12 19:04:18,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:18,059 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,059 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188688059 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:18,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:18,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:18,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:18,061 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:18,061 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:18,062 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:18,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188658062,5,FailOnTimeoutGroup] 2023-07-12 19:04:18,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188658062,5,FailOnTimeoutGroup] 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,063 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:18,125 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(951): ClusterId : b2bb5c2c-e1f7-476f-b609-451f2f518b82 2023-07-12 19:04:18,125 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:04:18,126 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:04:18,127 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:04:18,128 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:04:18,129 DEBUG [RS:0;jenkins-hbase4:36517] zookeeper.ReadOnlyZKClient(139): Connect 0x2d38ef84 to 127.0.0.1:54458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:18,134 DEBUG [RS:0;jenkins-hbase4:36517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27c084f4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:18,134 DEBUG [RS:0;jenkins-hbase4:36517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62c20e68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:18,143 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36517 2023-07-12 19:04:18,143 INFO [RS:0;jenkins-hbase4:36517] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:04:18,143 INFO [RS:0;jenkins-hbase4:36517] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:04:18,143 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:04:18,143 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,46015,1689188657003 with isa=jenkins-hbase4.apache.org/172.31.14.131:36517, startcode=1689188657060 2023-07-12 19:04:18,144 DEBUG [RS:0;jenkins-hbase4:36517] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:18,146 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46433, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:18,147 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46015] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,148 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868 2023-07-12 19:04:18,148 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:42335 2023-07-12 19:04:18,148 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:04:18,150 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:18,150 DEBUG [RS:0;jenkins-hbase4:36517] zookeeper.ZKUtil(162): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,150 WARN [RS:0;jenkins-hbase4:36517] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:18,150 INFO [RS:0;jenkins-hbase4:36517] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:18,150 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1948): logDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/WALs/jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,151 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36517,1689188657060] 2023-07-12 19:04:18,155 DEBUG [RS:0;jenkins-hbase4:36517] zookeeper.ZKUtil(162): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,156 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:04:18,156 INFO [RS:0;jenkins-hbase4:36517] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:04:18,157 INFO [RS:0;jenkins-hbase4:36517] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:04:18,158 INFO [RS:0;jenkins-hbase4:36517] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:04:18,158 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,158 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:04:18,159 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,159 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,160 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,160 DEBUG [RS:0;jenkins-hbase4:36517] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:18,160 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,160 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,160 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,171 INFO [RS:0;jenkins-hbase4:36517] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:04:18,171 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36517,1689188657060-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:18,185 INFO [RS:0;jenkins-hbase4:36517] regionserver.Replication(203): jenkins-hbase4.apache.org,36517,1689188657060 started 2023-07-12 19:04:18,185 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,36517,1689188657060, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36517, sessionid=0x1015aebe8ca0002 2023-07-12 19:04:18,185 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:04:18,185 DEBUG [RS:0;jenkins-hbase4:36517] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,185 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36517,1689188657060' 2023-07-12 19:04:18,185 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:04:18,185 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36517,1689188657060' 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:04:18,186 DEBUG [RS:0;jenkins-hbase4:36517] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:04:18,186 INFO [RS:0;jenkins-hbase4:36517] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 19:04:18,186 INFO [RS:0;jenkins-hbase4:36517] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 19:04:18,288 INFO [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36517%2C1689188657060, suffix=, logDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/WALs/jenkins-hbase4.apache.org,36517,1689188657060, archiveDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/oldWALs, maxLogs=32 2023-07-12 19:04:18,304 DEBUG [RS-EventLoopGroup-11-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK] 2023-07-12 19:04:18,307 INFO [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/WALs/jenkins-hbase4.apache.org,36517,1689188657060/jenkins-hbase4.apache.org%2C36517%2C1689188657060.1689188658289 2023-07-12 19:04:18,307 DEBUG [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK]] 2023-07-12 19:04:18,471 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:18,471 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:18,472 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868 2023-07-12 19:04:18,879 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:18,880 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:18,881 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/info 2023-07-12 19:04:18,882 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:18,882 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:18,883 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:18,884 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:18,884 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:18,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:18,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:18,886 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/table 2023-07-12 19:04:18,886 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:18,887 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:18,888 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740 2023-07-12 19:04:18,888 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740 2023-07-12 19:04:18,890 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:18,892 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:18,894 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67736263, jitterRate=0.009348973631858826}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:18,894 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:18,894 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:18,895 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:18,895 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:18,896 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:18,896 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:04:18,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:04:18,898 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:04:18,899 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:04:19,049 DEBUG [jenkins-hbase4:46015] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:04:19,050 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36517,1689188657060, state=OPENING 2023-07-12 19:04:19,052 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:04:19,053 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:19,053 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:19,053 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:19,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36517,1689188657060}] 2023-07-12 19:04:19,053 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:19,206 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:19,207 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:19,209 INFO [RS-EventLoopGroup-11-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:19,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:04:19,213 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:19,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36517%2C1689188657060.meta, suffix=.meta, logDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/WALs/jenkins-hbase4.apache.org,36517,1689188657060, archiveDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/oldWALs, maxLogs=32 2023-07-12 19:04:19,230 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK] 2023-07-12 19:04:19,232 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/WALs/jenkins-hbase4.apache.org,36517,1689188657060/jenkins-hbase4.apache.org%2C36517%2C1689188657060.meta.1689188659215.meta 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK]] 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:04:19,233 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:04:19,233 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:04:19,235 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:19,236 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/info 2023-07-12 19:04:19,236 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/info 2023-07-12 19:04:19,237 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:19,237 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:19,237 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:19,238 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:19,238 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:19,239 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:19,239 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:19,239 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:19,240 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/table 2023-07-12 19:04:19,240 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/table 2023-07-12 19:04:19,240 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:19,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:19,242 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740 2023-07-12 19:04:19,243 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740 2023-07-12 19:04:19,245 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:19,246 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:19,247 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73235866, jitterRate=0.09129944443702698}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:19,247 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:19,248 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689188659206 2023-07-12 19:04:19,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:04:19,252 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:04:19,252 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,36517,1689188657060, state=OPEN 2023-07-12 19:04:19,257 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:19,257 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:19,257 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:19,257 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:19,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:04:19,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,36517,1689188657060 in 204 msec 2023-07-12 19:04:19,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:04:19,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 363 msec 2023-07-12 19:04:19,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2050 sec 2023-07-12 19:04:19,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188659263, completionTime=-1 2023-07-12 19:04:19,263 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:19,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:19,265 DEBUG [hconnection-0xf2fe9a1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:19,267 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53220, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:19,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:19,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188719269 2023-07-12 19:04:19,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188779269 2023-07-12 19:04:19,269 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1689188657003-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1689188657003-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1689188657003-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46015, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:04:19,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:19,276 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:19,276 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:19,278 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:19,278 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:19,280 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:19,281 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c empty. 2023-07-12 19:04:19,281 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:19,281 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:04:19,691 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:19,692 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 92c77824bbc754096a5c7881b62f9c3c, NAME => 'hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp 2023-07-12 19:04:20,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:20,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 92c77824bbc754096a5c7881b62f9c3c, disabling compactions & flushes 2023-07-12 19:04:20,099 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. after waiting 0 ms 2023-07-12 19:04:20,099 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,099 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,100 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 92c77824bbc754096a5c7881b62f9c3c: 2023-07-12 19:04:20,102 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:20,103 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188660103"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188660103"}]},"ts":"1689188660103"} 2023-07-12 19:04:20,106 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:20,106 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:20,107 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188660107"}]},"ts":"1689188660107"} 2023-07-12 19:04:20,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:04:20,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=92c77824bbc754096a5c7881b62f9c3c, ASSIGN}] 2023-07-12 19:04:20,115 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=92c77824bbc754096a5c7881b62f9c3c, ASSIGN 2023-07-12 19:04:20,116 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=92c77824bbc754096a5c7881b62f9c3c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36517,1689188657060; forceNewPlan=false, retain=false 2023-07-12 19:04:20,266 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=92c77824bbc754096a5c7881b62f9c3c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:20,267 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188660266"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188660266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188660266"}]},"ts":"1689188660266"} 2023-07-12 19:04:20,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 92c77824bbc754096a5c7881b62f9c3c, server=jenkins-hbase4.apache.org,36517,1689188657060}] 2023-07-12 19:04:20,425 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,426 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 92c77824bbc754096a5c7881b62f9c3c, NAME => 'hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:20,426 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,426 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:20,426 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,426 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,427 INFO [StoreOpener-92c77824bbc754096a5c7881b62f9c3c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,429 DEBUG [StoreOpener-92c77824bbc754096a5c7881b62f9c3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/info 2023-07-12 19:04:20,429 DEBUG [StoreOpener-92c77824bbc754096a5c7881b62f9c3c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/info 2023-07-12 19:04:20,429 INFO [StoreOpener-92c77824bbc754096a5c7881b62f9c3c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 92c77824bbc754096a5c7881b62f9c3c columnFamilyName info 2023-07-12 19:04:20,430 INFO [StoreOpener-92c77824bbc754096a5c7881b62f9c3c-1] regionserver.HStore(310): Store=92c77824bbc754096a5c7881b62f9c3c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:20,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,434 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:20,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:20,436 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 92c77824bbc754096a5c7881b62f9c3c; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75346095, jitterRate=0.12274430692195892}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:20,436 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 92c77824bbc754096a5c7881b62f9c3c: 2023-07-12 19:04:20,438 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c., pid=6, masterSystemTime=1689188660421 2023-07-12 19:04:20,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,440 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:20,440 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:20,441 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188660440"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188660440"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188660440"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188660440"}]},"ts":"1689188660440"} 2023-07-12 19:04:20,444 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:04:20,444 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 92c77824bbc754096a5c7881b62f9c3c, server=jenkins-hbase4.apache.org,36517,1689188657060 in 173 msec 2023-07-12 19:04:20,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:04:20,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=92c77824bbc754096a5c7881b62f9c3c, ASSIGN in 332 msec 2023-07-12 19:04:20,447 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:20,447 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188660447"}]},"ts":"1689188660447"} 2023-07-12 19:04:20,449 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:04:20,451 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:20,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1760 sec 2023-07-12 19:04:20,478 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:04:20,479 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:20,479 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:20,479 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:20,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:20,491 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:20,494 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 10 msec 2023-07-12 19:04:20,506 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:20,513 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:20,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 10 msec 2023-07-12 19:04:20,531 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:20,533 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.402sec 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1689188657003-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:20,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46015,1689188657003-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:20,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:20,626 DEBUG [Listener at localhost/46683] zookeeper.ReadOnlyZKClient(139): Connect 0x5b76e8ba to 127.0.0.1:54458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:20,632 DEBUG [Listener at localhost/46683] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18554bcd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:20,634 DEBUG [hconnection-0xcecda65-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:20,636 INFO [RS-EventLoopGroup-11-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54766, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:20,638 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:20,638 WARN [Listener at localhost/46683] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:20,640 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:20,642 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:20,643 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:20,644 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:20,645 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 empty. 2023-07-12 19:04:20,645 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:20,646 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 19:04:21,056 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:21,057 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 434b98ec1d5027a4066750c8d9b30d17, NAME => 'testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testDeleteWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 434b98ec1d5027a4066750c8d9b30d17, disabling compactions & flushes 2023-07-12 19:04:21,464 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. after waiting 0 ms 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,464 INFO [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,464 DEBUG [RegionOpenAndInit-testDeleteWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 434b98ec1d5027a4066750c8d9b30d17: 2023-07-12 19:04:21,467 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:21,468 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188661468"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188661468"}]},"ts":"1689188661468"} 2023-07-12 19:04:21,470 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:21,471 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:21,471 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188661471"}]},"ts":"1689188661471"} 2023-07-12 19:04:21,472 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 19:04:21,475 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, ASSIGN}] 2023-07-12 19:04:21,477 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, ASSIGN 2023-07-12 19:04:21,477 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36517,1689188657060; forceNewPlan=false, retain=false 2023-07-12 19:04:21,628 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=434b98ec1d5027a4066750c8d9b30d17, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:21,629 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188661628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188661628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188661628"}]},"ts":"1689188661628"} 2023-07-12 19:04:21,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 434b98ec1d5027a4066750c8d9b30d17, server=jenkins-hbase4.apache.org,36517,1689188657060}] 2023-07-12 19:04:21,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 434b98ec1d5027a4066750c8d9b30d17, NAME => 'testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:21,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testDeleteWithFailoverAtStep4 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:21,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,790 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,791 DEBUG [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f1 2023-07-12 19:04:21,791 DEBUG [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f1 2023-07-12 19:04:21,792 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 434b98ec1d5027a4066750c8d9b30d17 columnFamilyName f1 2023-07-12 19:04:21,793 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] regionserver.HStore(310): Store=434b98ec1d5027a4066750c8d9b30d17/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:21,793 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,794 DEBUG [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f2 2023-07-12 19:04:21,794 DEBUG [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f2 2023-07-12 19:04:21,799 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 434b98ec1d5027a4066750c8d9b30d17 columnFamilyName f2 2023-07-12 19:04:21,799 INFO [StoreOpener-434b98ec1d5027a4066750c8d9b30d17-1] regionserver.HStore(310): Store=434b98ec1d5027a4066750c8d9b30d17/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:21,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testDeleteWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:21,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:21,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:21,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 434b98ec1d5027a4066750c8d9b30d17; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62581214, jitterRate=-0.06746724247932434}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:21,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 434b98ec1d5027a4066750c8d9b30d17: 2023-07-12 19:04:21,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17., pid=11, masterSystemTime=1689188661783 2023-07-12 19:04:21,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:21,811 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=434b98ec1d5027a4066750c8d9b30d17, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:21,811 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188661811"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188661811"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188661811"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188661811"}]},"ts":"1689188661811"} 2023-07-12 19:04:21,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-12 19:04:21,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 434b98ec1d5027a4066750c8d9b30d17, server=jenkins-hbase4.apache.org,36517,1689188657060 in 182 msec 2023-07-12 19:04:21,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 19:04:21,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, ASSIGN in 341 msec 2023-07-12 19:04:21,818 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:21,819 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188661818"}]},"ts":"1689188661818"} 2023-07-12 19:04:21,820 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 19:04:21,822 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testDeleteWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:21,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 in 1.1830 sec 2023-07-12 19:04:21,892 DEBUG [Listener at localhost/46683] util.CommonFSUtils(682): File system contents for path hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4 2023-07-12 19:04:21,892 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 19:04:21,893 DEBUG [Listener at localhost/46683] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 19:04:21,893 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 19:04:21,893 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |-434b98ec1d5027a4066750c8d9b30d17/ 2023-07-12 19:04:21,894 DEBUG [Listener at localhost/46683] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:21,894 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:21,894 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:21,895 DEBUG [Listener at localhost/46683] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:21,895 DEBUG [Listener at localhost/46683] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:21,898 DEBUG [Listener at localhost/46683] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 19:04:21,912 DEBUG [Listener at localhost/46683] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 19:04:21,915 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:32924, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 19:04:21,920 INFO [Listener at localhost/46683] client.HBaseAdmin$15(890): Started disable of testDeleteWithFailoverAtStep4 2023-07-12 19:04:21,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testDeleteWithFailoverAtStep4 2023-07-12 19:04:21,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:21,928 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188661928"}]},"ts":"1689188661928"} 2023-07-12 19:04:21,930 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-12 19:04:21,933 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testDeleteWithFailoverAtStep4 to state=DISABLING 2023-07-12 19:04:21,934 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, UNASSIGN}] 2023-07-12 19:04:21,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:04:21,935 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, UNASSIGN 2023-07-12 19:04:21,935 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=434b98ec1d5027a4066750c8d9b30d17, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:21,936 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188661935"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188661935"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188661935"}]},"ts":"1689188661935"} 2023-07-12 19:04:21,937 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 434b98ec1d5027a4066750c8d9b30d17, server=jenkins-hbase4.apache.org,36517,1689188657060}] 2023-07-12 19:04:22,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:04:22,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 434b98ec1d5027a4066750c8d9b30d17, disabling compactions & flushes 2023-07-12 19:04:22,090 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:22,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:22,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. after waiting 0 ms 2023-07-12 19:04:22,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:22,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:22,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17. 2023-07-12 19:04:22,095 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 434b98ec1d5027a4066750c8d9b30d17: 2023-07-12 19:04:22,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,097 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=434b98ec1d5027a4066750c8d9b30d17, regionState=CLOSED 2023-07-12 19:04:22,097 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testDeleteWithFailoverAtStep4,,1689188660639.434b98ec1d5027a4066750c8d9b30d17.","families":{"info":[{"qualifier":"regioninfo","vlen":63,"tag":[],"timestamp":"1689188662097"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188662097"}]},"ts":"1689188662097"} 2023-07-12 19:04:22,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 19:04:22,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 434b98ec1d5027a4066750c8d9b30d17, server=jenkins-hbase4.apache.org,36517,1689188657060 in 162 msec 2023-07-12 19:04:22,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-12 19:04:22,102 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testDeleteWithFailoverAtStep4, region=434b98ec1d5027a4066750c8d9b30d17, UNASSIGN in 167 msec 2023-07-12 19:04:22,103 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188662103"}]},"ts":"1689188662103"} 2023-07-12 19:04:22,105 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testDeleteWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-12 19:04:22,106 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testDeleteWithFailoverAtStep4 to state=DISABLED 2023-07-12 19:04:22,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 in 183 msec 2023-07-12 19:04:22,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46015] master.MasterRpcServices(1230): Checking to see if procedure is done pid=12 2023-07-12 19:04:22,241 INFO [Listener at localhost/46683] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testDeleteWithFailoverAtStep4, procId: 12 completed 2023-07-12 19:04:22,241 WARN [Listener at localhost/46683] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 19:04:22,246 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,248 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,249 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:22,249 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,249 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:22,249 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:22,497 INFO [Listener at localhost/46683] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,497 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:22,497 INFO [Listener at localhost/46683] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:22,497 INFO [Listener at localhost/46683] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:22,498 ERROR [Listener at localhost/46683] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:22,499 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:22,499 INFO [Listener at localhost/46683] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:22,499 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:22,499 INFO [Listener at localhost/46683] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:22,500 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:22,501 DEBUG [Listener at localhost/46683] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[] 2023-07-12 19:04:22,501 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:22,501 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:22,502 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:22,502 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:22,502 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,502 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,502 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,503 DEBUG [Listener at localhost/46683] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:22,503 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:22,503 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:22,504 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36517,1689188657060, table=hbase:meta, region=1588230740 2023-07-12 19:04:22,504 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188692504 2023-07-12 19:04:22,505 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,505 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:22,508 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:22,508 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=434b98ec1d5027a4066750c8d9b30d17, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:22,508 INFO [Listener at localhost/46683] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:22,508 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188722508 2023-07-12 19:04:22,508 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188782508 2023-07-12 19:04:22,508 INFO [Listener at localhost/46683] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:22,508 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,508 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:22,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,514 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,516 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f1, FileablePath, hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/f2, FileablePath, hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/recovered.edits] 2023-07-12 19:04:22,522 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/recovered.edits/4.seqid to hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/archive/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17/recovered.edits/4.seqid 2023-07-12 19:04:22,523 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,523 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 19:04:22,524 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:22,525 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:22,525 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:22,758 INFO [Listener at localhost/46683] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:22,759 ERROR [Listener at localhost/46683] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:22,759 INFO [Listener at localhost/46683] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:22,760 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:22,761 DEBUG [Listener at localhost/46683] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0] 2023-07-12 19:04:22,761 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:22,762 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:22,762 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:22,762 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:22,763 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,763 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,763 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,763 DEBUG [Listener at localhost/46683] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:22,763 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 19:04:22,764 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:22,764 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36517,1689188657060, table=hbase:meta, region=1588230740 2023-07-12 19:04:22,765 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188692765 2023-07-12 19:04:22,766 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:22,766 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,768 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:22,769 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=434b98ec1d5027a4066750c8d9b30d17, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:22,769 INFO [Listener at localhost/46683] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:22,769 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188722769 2023-07-12 19:04:22,769 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188782769 2023-07-12 19:04:22,769 INFO [Listener at localhost/46683] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:22,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,771 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,772 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 empty. 2023-07-12 19:04:22,772 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.tmp/data/default/testDeleteWithFailoverAtStep4/434b98ec1d5027a4066750c8d9b30d17 2023-07-12 19:04:22,772 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testDeleteWithFailoverAtStep4 regions 2023-07-12 19:04:22,774 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:22,775 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,796 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testDeleteWithFailoverAtStep4 from hbase:meta 2023-07-12 19:04:22,799 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-12 19:04:22,799 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:22,799 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:22,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:22,799 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:23,019 INFO [Listener at localhost/46683] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,019 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:23,019 INFO [Listener at localhost/46683] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:23,019 INFO [Listener at localhost/46683] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:23,020 ERROR [Listener at localhost/46683] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:23,020 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:23,020 INFO [Listener at localhost/46683] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:23,020 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:23,020 INFO [Listener at localhost/46683] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:23,021 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:23,021 DEBUG [Listener at localhost/46683] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1] 2023-07-12 19:04:23,022 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:23,022 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:23,022 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:23,023 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:23,023 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,023 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,023 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,023 DEBUG [Listener at localhost/46683] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:23,024 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:23,024 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:23,025 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36517,1689188657060, table=hbase:meta, region=1588230740 2023-07-12 19:04:23,025 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188693025 2023-07-12 19:04:23,026 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:23,026 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,029 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:23,029 INFO [Listener at localhost/46683] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:23,030 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188723030 2023-07-12 19:04:23,030 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188783030 2023-07-12 19:04:23,030 INFO [Listener at localhost/46683] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 19:04:23,030 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,032 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testDeleteWithFailoverAtStep4' descriptor. 2023-07-12 19:04:23,032 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:23,034 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,034 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-12 19:04:23,034 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 19:04:23,034 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 19:04:23,034 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-12 19:04:23,034 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689188663034"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:23,037 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-12 19:04:23,038 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:23,038 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,038 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:23,038 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:23,280 INFO [Listener at localhost/46683] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,280 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:23,280 INFO [Listener at localhost/46683] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:23,280 INFO [Listener at localhost/46683] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:23,281 ERROR [Listener at localhost/46683] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:23,281 INFO [Listener at localhost/46683] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:23,281 INFO [Listener at localhost/46683] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:23,281 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:23,281 INFO [Listener at localhost/46683] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:23,282 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:23,283 DEBUG [Listener at localhost/46683] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2] 2023-07-12 19:04:23,283 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:23,284 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:23,284 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:23,284 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:23,284 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,285 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,285 DEBUG [Listener at localhost/46683] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,285 DEBUG [Listener at localhost/46683] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:23,285 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:23,286 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:23,286 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36517,1689188657060, table=hbase:meta, region=1588230740 2023-07-12 19:04:23,286 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188693286 2023-07-12 19:04:23,287 DEBUG [Listener at localhost/46683] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:23,287 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,289 DEBUG [Listener at localhost/46683] assignment.RegionStateStore(147): Load hbase:meta entry region=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:23,290 INFO [Listener at localhost/46683] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:23,290 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188723290 2023-07-12 19:04:23,290 INFO [Listener at localhost/46683] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188783290 2023-07-12 19:04:23,290 INFO [Listener at localhost/46683] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 19:04:23,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testDeleteWithFailoverAtStep4' from region states. 2023-07-12 19:04:23,290 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 19:04:23,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 19:04:23,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testDeleteWithFailoverAtStep4' as deleted. 2023-07-12 19:04:23,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testDeleteWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689188663290"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:23,293 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testDeleteWithFailoverAtStep4 state from META 2023-07-12 19:04:23,294 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testDeleteWithFailoverAtStep4 already deleted, retry=false 2023-07-12 19:04:23,294 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:23,295 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,295 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:23,295 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:23,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:23,295 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=15, state=SUCCESS, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:23,540 INFO [Listener at localhost/46683] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 19:04:23,540 INFO [Listener at localhost/46683] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:23,540 ERROR [Listener at localhost/46683] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 19:04:23,540 ERROR [Listener at localhost/46683] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase4.apache.org,46015,1689188657003: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailoverAtStep(TestMasterFailoverWithProcedures.java:159) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testDeleteWithFailover(TestMasterFailoverWithProcedures.java:137) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:23,540 INFO [Listener at localhost/46683] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,46015,1689188657003' ***** 2023-07-12 19:04:23,541 INFO [Listener at localhost/46683] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/46683 2023-07-12 19:04:23,541 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bacb3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:23,542 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,46015,1689188657003 2023-07-12 19:04:23,542 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07055ccf to 127.0.0.1:54458 2023-07-12 19:04:23,542 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:23,542 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,46015,1689188657003; all regions closed. 2023-07-12 19:04:23,542 DEBUG [M:0;jenkins-hbase4:46015] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:23,542 DEBUG [M:0;jenkins-hbase4:46015] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:23,542 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:23,542 DEBUG [M:0;jenkins-hbase4:46015] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:23,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188658062] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188658062,5,FailOnTimeoutGroup] 2023-07-12 19:04:23,543 INFO [M:0;jenkins-hbase4:46015] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:23,542 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188658062] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188658062,5,FailOnTimeoutGroup] 2023-07-12 19:04:23,543 INFO [M:0;jenkins-hbase4:46015] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:23,544 INFO [M:0;jenkins-hbase4:46015] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:23,544 DEBUG [M:0;jenkins-hbase4:46015] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:23,544 INFO [M:0;jenkins-hbase4:46015] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:23,545 ERROR [M:0;jenkins-hbase4:46015] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:23,545 INFO [M:0;jenkins-hbase4:46015] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 19:04:23,545 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:23,545 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,545 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,545 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:23,545 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:23,545 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,545 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.ZKUtil(398): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:23,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:23,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:23,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:23,545 WARN [M:0;jenkins-hbase4:46015] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:23,546 INFO [M:0;jenkins-hbase4:46015] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:23,547 INFO [M:0;jenkins-hbase4:46015] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 19:04:23,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:23,547 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 19:04:23,548 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,549 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,549 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:23,548 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:23,549 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:23,549 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:23,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42421,1689188657030 from backup master directory 2023-07-12 19:04:23,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:23,550 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:23,551 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:23,551 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:23,551 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:23,551 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:23,551 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:23,551 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:23,551 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:23,551 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:23,551 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:23,551 ERROR [M:0;jenkins-hbase4:46015] regionserver.HRegion(1825): Memstore data size is 52368 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:23,551 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:23,551 DEBUG [M:0;jenkins-hbase4:46015] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:23,551 INFO [M:0;jenkins-hbase4:46015] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:23,551 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:23,552 INFO [M:0;jenkins-hbase4:46015] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46015 2023-07-12 19:04:23,555 DEBUG [M:0;jenkins-hbase4:46015] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46015,1689188657003 already deleted, retry=false 2023-07-12 19:04:23,566 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:23,587 WARN [RS-EventLoopGroup-11-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:46015 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46015 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:23,588 DEBUG [RS-EventLoopGroup-11-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:46015 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:46015 2023-07-12 19:04:23,688 DEBUG [RS:0;jenkins-hbase4:36517] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:23,691 INFO [RS-EventLoopGroup-10-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51145, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:23,691 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:23,746 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:23,746 INFO [M:0;jenkins-hbase4:46015] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,46015,1689188657003; zookeeper connection closed. 2023-07-12 19:04:23,746 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:46015-0x1015aebe8ca0000, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:23,794 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:23,897 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:23,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7ab26c44 to 127.0.0.1:54458 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:23,984 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71170744, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:23,984 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:23,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:23,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:23,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003 to hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003-dead as it is dead 2023-07-12 19:04:23,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003-dead/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:23,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003-dead/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 after 0ms 2023-07-12 19:04:23,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003-dead/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 to hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:23,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,46015,1689188657003-dead 2023-07-12 19:04:23,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:23,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42421%2C1689188657030, suffix=, logDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,42421,1689188657030, archiveDir=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:23,999 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:24,010 DEBUG [RS-EventLoopGroup-11-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK] 2023-07-12 19:04:24,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/WALs/jenkins-hbase4.apache.org,42421,1689188657030/jenkins-hbase4.apache.org%2C42421%2C1689188657030.1689188663995 2023-07-12 19:04:24,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39295,DS-c716bf2a-1aea-4f65-aea9-bec7d52e8618,DISK]] 2023-07-12 19:04:24,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:24,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:24,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:24,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:24,014 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:24,014 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:24,015 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:24,015 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:24,016 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 19:04:24,016 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:24,022 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 133, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=119, path=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:24,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=51.14 KB heapSize=61.23 KB 2023-07-12 19:04:24,101 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:24,203 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:24,305 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:24,408 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42421] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:24,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.14 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/17da17215d4e4c118cae2fac50ab8666 2023-07-12 19:04:24,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/17da17215d4e4c118cae2fac50ab8666 as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/17da17215d4e4c118cae2fac50ab8666 2023-07-12 19:04:24,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/17da17215d4e4c118cae2fac50ab8666, entries=133, sequenceid=119, filesize=56.0 K 2023-07-12 19:04:24,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~51.14 KB/52368, heapSize ~61.21 KB/62680, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 421ms, sequenceid=119, compaction requested=false; wal=null 2023-07-12 19:04:24,444 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C46015%2C1689188657003.1689188657986 2023-07-12 19:04:24,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:24,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/119.seqid, newMaxSeqId=119, maxSeqId=1 2023-07-12 19:04:24,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=120; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73098278, jitterRate=0.08924922347068787}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:24,450 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:24,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:24,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:24,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:24,451 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:24,452 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:24,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=15, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-12 19:04:24,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:24,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:24,455 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:24,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:24,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:24,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; DisableTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:24,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:24,456 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.Procedure(966): pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:24,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-12 19:04:24,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:24,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:24,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,36517,1689188657060, table=hbase:meta, region=1588230740 2023-07-12 19:04:24,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:24,459 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:24,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:24,470 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:24,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:24,471 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:24,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:24,472 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:24,472 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:24,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:24,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42421,1689188657030, sessionid=0x1015aebe8ca0001, setting cluster-up flag (Was=true) 2023-07-12 19:04:24,475 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:24,476 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:24,478 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:24,478 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:24,479 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/.hbase-snapshot/.tmp 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:24,480 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:24,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188694480 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:24,481 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:24,481 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:24,482 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:24,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188664482,5,FailOnTimeoutGroup] 2023-07-12 19:04:24,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188664483,5,FailOnTimeoutGroup] 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188664483, completionTime=-1 2023-07-12 19:04:24,483 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:24,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:24,484 DEBUG [hconnection-0x29d39c35-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:24,486 INFO [RS-EventLoopGroup-11-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:24,488 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=92c77824bbc754096a5c7881b62f9c3c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36517,1689188657060, regionLocation=jenkins-hbase4.apache.org,36517,1689188657060, openSeqNum=2 2023-07-12 19:04:24,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:24,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188724488 2023-07-12 19:04:24,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188784488 2023-07-12 19:04:24,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 19:04:24,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42421,1689188657030-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42421,1689188657030-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42421,1689188657030-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42421, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,495 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:24,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 19:04:24,497 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:24,506 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:24,509 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:24,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.958sec 2023-07-12 19:04:24,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:24,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:24,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:24,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42421,1689188657030-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:24,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42421,1689188657030-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:24,512 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:24,560 DEBUG [PEWorker-1] procedure.ProcedureSyncWait(216): waitFor quota manager to be available 2023-07-12 19:04:24,560 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 2023-07-12 19:04:24,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=testDeleteWithFailoverAtStep4 in 2.3190 sec 2023-07-12 19:04:24,735 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:24,799 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:04:24,800 INFO [Listener at localhost/46683] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 19:04:24,800 DEBUG [Listener at localhost/46683] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b76e8ba to 127.0.0.1:54458 2023-07-12 19:04:24,800 DEBUG [Listener at localhost/46683] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:24,800 DEBUG [Listener at localhost/46683] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:04:24,800 DEBUG [Listener at localhost/46683] util.JVMClusterUtil(257): Found active master hash=1740400262, stopped=false 2023-07-12 19:04:24,800 INFO [Listener at localhost/46683] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:24,804 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:24,804 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:24,804 INFO [Listener at localhost/46683] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:24,804 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:24,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:24,806 DEBUG [Listener at localhost/46683] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7ab26c44 to 127.0.0.1:54458 2023-07-12 19:04:24,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:24,806 DEBUG [Listener at localhost/46683] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:24,806 INFO [Listener at localhost/46683] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,36517,1689188657060' ***** 2023-07-12 19:04:24,806 INFO [Listener at localhost/46683] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:04:24,806 INFO [RS:0;jenkins-hbase4:36517] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:04:24,806 INFO [RS:0;jenkins-hbase4:36517] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:04:24,806 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:04:24,806 INFO [RS:0;jenkins-hbase4:36517] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:04:24,807 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(3305): Received CLOSE for 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:24,807 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:24,807 DEBUG [RS:0;jenkins-hbase4:36517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d38ef84 to 127.0.0.1:54458 2023-07-12 19:04:24,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 92c77824bbc754096a5c7881b62f9c3c, disabling compactions & flushes 2023-07-12 19:04:24,808 DEBUG [RS:0;jenkins-hbase4:36517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:24,808 INFO [RS:0;jenkins-hbase4:36517] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:04:24,808 INFO [RS:0;jenkins-hbase4:36517] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:04:24,808 INFO [RS:0;jenkins-hbase4:36517] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:04:24,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:24,808 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:04:24,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:24,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. after waiting 0 ms 2023-07-12 19:04:24,808 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:24,808 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 92c77824bbc754096a5c7881b62f9c3c 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 19:04:24,809 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-12 19:04:24,809 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1478): Online Regions={92c77824bbc754096a5c7881b62f9c3c=hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c., 1588230740=hbase:meta,,1.1588230740} 2023-07-12 19:04:24,809 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740, 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:24,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:24,809 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:24,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:24,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:24,809 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:24,809 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.00 KB heapSize=7.39 KB 2023-07-12 19:04:24,865 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:25,009 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740, 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:25,209 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740, 92c77824bbc754096a5c7881b62f9c3c 2023-07-12 19:04:25,218 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/.tmp/info/5f10c2968d694f36981593ba0dd653f8 2023-07-12 19:04:25,218 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.35 KB at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/info/118d57070048488d93d5d25dbc6b8cc0 2023-07-12 19:04:25,224 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 118d57070048488d93d5d25dbc6b8cc0 2023-07-12 19:04:25,227 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/.tmp/info/5f10c2968d694f36981593ba0dd653f8 as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/info/5f10c2968d694f36981593ba0dd653f8 2023-07-12 19:04:25,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/info/5f10c2968d694f36981593ba0dd653f8, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 19:04:25,235 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 92c77824bbc754096a5c7881b62f9c3c in 427ms, sequenceid=6, compaction requested=false 2023-07-12 19:04:25,239 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/namespace/92c77824bbc754096a5c7881b62f9c3c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 19:04:25,240 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:25,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 92c77824bbc754096a5c7881b62f9c3c: 2023-07-12 19:04:25,240 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689188659275.92c77824bbc754096a5c7881b62f9c3c. 2023-07-12 19:04:25,250 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 19:04:25,250 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 19:04:25,409 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:25,610 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:25,633 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=109 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/rep_barrier/5d3ccbb94fdd47238f5876268f5c37ea 2023-07-12 19:04:25,639 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5d3ccbb94fdd47238f5876268f5c37ea 2023-07-12 19:04:25,810 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 19:04:25,810 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-07-12 19:04:25,810 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:25,987 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:25,987 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 19:04:26,010 DEBUG [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:26,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=559 B at sequenceid=21 (bloomFilter=false), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/table/985bffe7c4a042c78fca29bc2f2d090b 2023-07-12 19:04:26,053 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 985bffe7c4a042c78fca29bc2f2d090b 2023-07-12 19:04:26,054 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/info/118d57070048488d93d5d25dbc6b8cc0 as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/info/118d57070048488d93d5d25dbc6b8cc0 2023-07-12 19:04:26,059 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 118d57070048488d93d5d25dbc6b8cc0 2023-07-12 19:04:26,059 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/info/118d57070048488d93d5d25dbc6b8cc0, entries=11, sequenceid=21, filesize=6.2 K 2023-07-12 19:04:26,060 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/rep_barrier/5d3ccbb94fdd47238f5876268f5c37ea as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/rep_barrier/5d3ccbb94fdd47238f5876268f5c37ea 2023-07-12 19:04:26,066 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5d3ccbb94fdd47238f5876268f5c37ea 2023-07-12 19:04:26,066 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/rep_barrier/5d3ccbb94fdd47238f5876268f5c37ea, entries=1, sequenceid=21, filesize=5.0 K 2023-07-12 19:04:26,067 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/.tmp/table/985bffe7c4a042c78fca29bc2f2d090b as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/table/985bffe7c4a042c78fca29bc2f2d090b 2023-07-12 19:04:26,072 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 985bffe7c4a042c78fca29bc2f2d090b 2023-07-12 19:04:26,072 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/table/985bffe7c4a042c78fca29bc2f2d090b, entries=5, sequenceid=21, filesize=5.1 K 2023-07-12 19:04:26,073 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.00 KB/4094, heapSize ~7.34 KB/7520, currentSize=0 B/0 for 1588230740 in 1264ms, sequenceid=21, compaction requested=false 2023-07-12 19:04:26,079 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/data/hbase/meta/1588230740/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=1 2023-07-12 19:04:26,080 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:26,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:26,080 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:26,080 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:26,211 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,36517,1689188657060; all regions closed. 2023-07-12 19:04:26,216 DEBUG [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/oldWALs 2023-07-12 19:04:26,216 INFO [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36517%2C1689188657060.meta:.meta(num 1689188659215) 2023-07-12 19:04:26,220 DEBUG [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/oldWALs 2023-07-12 19:04:26,220 INFO [RS:0;jenkins-hbase4:36517] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36517%2C1689188657060:(num 1689188658289) 2023-07-12 19:04:26,221 DEBUG [RS:0;jenkins-hbase4:36517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:26,221 INFO [RS:0;jenkins-hbase4:36517] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:26,221 INFO [RS:0;jenkins-hbase4:36517] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 19:04:26,221 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:26,222 INFO [RS:0;jenkins-hbase4:36517] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36517 2023-07-12 19:04:26,224 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:26,224 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36517,1689188657060 2023-07-12 19:04:26,224 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:26,226 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36517,1689188657060] 2023-07-12 19:04:26,226 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36517,1689188657060; numProcessing=1 2023-07-12 19:04:26,230 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36517,1689188657060 already deleted, retry=false 2023-07-12 19:04:26,230 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36517,1689188657060 expired; onlineServers=0 2023-07-12 19:04:26,230 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42421,1689188657030' ***** 2023-07-12 19:04:26,230 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:04:26,230 DEBUG [M:1;jenkins-hbase4:42421] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6383eea9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:26,230 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,42421,1689188657030 2023-07-12 19:04:26,230 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42421,1689188657030; all regions closed. 2023-07-12 19:04:26,230 DEBUG [M:1;jenkins-hbase4:42421] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:26,230 DEBUG [M:1;jenkins-hbase4:42421] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:26,231 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:26,231 DEBUG [M:1;jenkins-hbase4:42421] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:26,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188664483] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188664483,5,FailOnTimeoutGroup] 2023-07-12 19:04:26,231 INFO [M:1;jenkins-hbase4:42421] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:26,231 INFO [M:1;jenkins-hbase4:42421] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:26,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188664482] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188664482,5,FailOnTimeoutGroup] 2023-07-12 19:04:26,231 INFO [M:1;jenkins-hbase4:42421] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:26,231 DEBUG [M:1;jenkins-hbase4:42421] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:26,231 INFO [M:1;jenkins-hbase4:42421] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:26,231 ERROR [M:1;jenkins-hbase4:42421] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:26,231 INFO [M:1;jenkins-hbase4:42421] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:26,231 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:26,233 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:26,233 DEBUG [M:1;jenkins-hbase4:42421] zookeeper.ZKUtil(398): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:26,233 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:26,233 WARN [M:1;jenkins-hbase4:42421] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:26,233 INFO [M:1;jenkins-hbase4:42421] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:26,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:26,234 INFO [M:1;jenkins-hbase4:42421] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:04:26,234 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:26,234 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:26,234 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:26,234 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:26,234 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:26,235 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=661 B heapSize=1.05 KB 2023-07-12 19:04:26,326 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:26,326 INFO [RS:0;jenkins-hbase4:36517] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,36517,1689188657060; zookeeper connection closed. 2023-07-12 19:04:26,326 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): regionserver:36517-0x1015aebe8ca0002, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:26,328 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7c8f296] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7c8f296 2023-07-12 19:04:26,328 INFO [Listener at localhost/46683] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 19:04:26,644 INFO [M:1;jenkins-hbase4:42421] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=661 B at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/42675b1250de421ebae898eee47878ec 2023-07-12 19:04:26,650 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/42675b1250de421ebae898eee47878ec as hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/42675b1250de421ebae898eee47878ec 2023-07-12 19:04:26,655 INFO [M:1;jenkins-hbase4:42421] regionserver.HStore(1080): Added hdfs://localhost:42335/user/jenkins/test-data/1b6bdd51-8fa8-749e-6d40-1f103a7de868/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/42675b1250de421ebae898eee47878ec, entries=1, sequenceid=123, filesize=5.1 K 2023-07-12 19:04:26,656 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegion(2948): Finished flush of dataSize ~661 B/661, heapSize ~1.03 KB/1056, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 422ms, sequenceid=123, compaction requested=false 2023-07-12 19:04:26,658 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:26,658 DEBUG [M:1;jenkins-hbase4:42421] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:26,662 INFO [M:1;jenkins-hbase4:42421] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:26,662 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:26,662 INFO [M:1;jenkins-hbase4:42421] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42421 2023-07-12 19:04:26,664 DEBUG [M:1;jenkins-hbase4:42421] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42421,1689188657030 already deleted, retry=false 2023-07-12 19:04:26,765 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:26,765 INFO [M:1;jenkins-hbase4:42421] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42421,1689188657030; zookeeper connection closed. 2023-07-12 19:04:26,765 DEBUG [Listener at localhost/46683-EventThread] zookeeper.ZKWatcher(600): master:42421-0x1015aebe8ca0001, quorum=127.0.0.1:54458, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:26,766 WARN [Listener at localhost/46683] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:04:26,770 INFO [Listener at localhost/46683] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:26,873 WARN [BP-796178925-172.31.14.131-1689188656116 heartbeating to localhost/127.0.0.1:42335] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:04:26,874 WARN [BP-796178925-172.31.14.131-1689188656116 heartbeating to localhost/127.0.0.1:42335] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-796178925-172.31.14.131-1689188656116 (Datanode Uuid 269d8c42-ef61-4d65-8c79-3ccfe886cc5d) service to localhost/127.0.0.1:42335 2023-07-12 19:04:26,874 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17/dfs/data/data1/current/BP-796178925-172.31.14.131-1689188656116] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:26,875 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/cluster_dade0f3d-bcb2-cc03-c912-d267598f7c17/dfs/data/data2/current/BP-796178925-172.31.14.131-1689188656116] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:26,886 INFO [Listener at localhost/46683] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:26,995 INFO [Listener at localhost/46683] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:04:27,005 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 19:04:27,015 INFO [Listener at localhost/46683] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testDeleteWithFailover Thread=102 (was 84) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:42335 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: Listener at localhost/46683 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:42335 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:42335 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:42335 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-11-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:42335 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=443 (was 420) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=273 (was 244) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=8348 (was 8530) 2023-07-12 19:04:27,023 INFO [Listener at localhost/46683] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=102, OpenFileDescriptor=443, MaxFileDescriptor=60000, SystemLoadAverage=273, ProcessCount=170, AvailableMemoryMB=8349 2023-07-12 19:04:27,023 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:04:27,023 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/hadoop.log.dir so I do NOT create it in target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288 2023-07-12 19:04:27,023 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/8bf53dc1-ef88-6bda-33e0-6a1e57f1fb54/hadoop.tmp.dir so I do NOT create it in target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c, deleteOnExit=true 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/test.cache.data in system properties and HBase conf 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:04:27,024 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:04:27,024 DEBUG [Listener at localhost/46683] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:04:27,025 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:04:27,026 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:04:27,026 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:27,026 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:04:27,026 INFO [Listener at localhost/46683] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:04:27,030 WARN [Listener at localhost/46683] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:27,030 WARN [Listener at localhost/46683] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:27,072 WARN [Listener at localhost/46683] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:27,074 INFO [Listener at localhost/46683] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:27,079 INFO [Listener at localhost/46683] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/java.io.tmpdir/Jetty_localhost_40031_hdfs____4wuipo/webapp 2023-07-12 19:04:27,171 INFO [Listener at localhost/46683] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40031 2023-07-12 19:04:27,175 WARN [Listener at localhost/46683] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:27,176 WARN [Listener at localhost/46683] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:27,213 WARN [Listener at localhost/40837] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:27,228 WARN [Listener at localhost/40837] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:04:27,230 WARN [Listener at localhost/40837] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:27,231 INFO [Listener at localhost/40837] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:27,236 INFO [Listener at localhost/40837] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/java.io.tmpdir/Jetty_localhost_36583_datanode____.goovzg/webapp 2023-07-12 19:04:27,330 INFO [Listener at localhost/40837] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36583 2023-07-12 19:04:27,337 WARN [Listener at localhost/34615] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:27,426 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1bd1dd7768a82674: Processing first storage report for DS-b79f8327-1c99-4b46-ba1f-ee5250908e62 from datanode 9642a675-1733-4f54-9789-2d4949d4bae8 2023-07-12 19:04:27,426 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bd1dd7768a82674: from storage DS-b79f8327-1c99-4b46-ba1f-ee5250908e62 node DatanodeRegistration(127.0.0.1:35881, datanodeUuid=9642a675-1733-4f54-9789-2d4949d4bae8, infoPort=43795, infoSecurePort=0, ipcPort=34615, storageInfo=lv=-57;cid=testClusterID;nsid=1941602023;c=1689188667032), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:27,426 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1bd1dd7768a82674: Processing first storage report for DS-3cb6b2d0-7c23-46fd-8897-fe670ca21840 from datanode 9642a675-1733-4f54-9789-2d4949d4bae8 2023-07-12 19:04:27,426 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bd1dd7768a82674: from storage DS-3cb6b2d0-7c23-46fd-8897-fe670ca21840 node DatanodeRegistration(127.0.0.1:35881, datanodeUuid=9642a675-1733-4f54-9789-2d4949d4bae8, infoPort=43795, infoSecurePort=0, ipcPort=34615, storageInfo=lv=-57;cid=testClusterID;nsid=1941602023;c=1689188667032), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:27,443 DEBUG [Listener at localhost/34615] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288 2023-07-12 19:04:27,445 INFO [Listener at localhost/34615] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c/zookeeper_0, clientPort=61051, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:04:27,446 INFO [Listener at localhost/34615] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=61051 2023-07-12 19:04:27,447 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,447 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,857 INFO [Listener at localhost/34615] util.FSUtils(471): Created version file at hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589 with version=8 2023-07-12 19:04:27,857 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase-staging 2023-07-12 19:04:27,859 INFO [Listener at localhost/34615] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:27,859 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,859 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,859 INFO [Listener at localhost/34615] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:27,860 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,860 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:27,860 INFO [Listener at localhost/34615] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:27,861 INFO [Listener at localhost/34615] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38203 2023-07-12 19:04:27,862 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,863 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,864 INFO [Listener at localhost/34615] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38203 connecting to ZooKeeper ensemble=127.0.0.1:61051 2023-07-12 19:04:27,870 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:382030x0, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:27,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38203-0x1015aec13380000 connected 2023-07-12 19:04:27,884 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:27,884 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:27,885 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:27,885 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38203 2023-07-12 19:04:27,885 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38203 2023-07-12 19:04:27,885 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38203 2023-07-12 19:04:27,886 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38203 2023-07-12 19:04:27,886 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38203 2023-07-12 19:04:27,886 INFO [Listener at localhost/34615] master.HMaster(444): hbase.rootdir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589, hbase.cluster.distributed=false 2023-07-12 19:04:27,887 INFO [Listener at localhost/34615] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:27,888 INFO [Listener at localhost/34615] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:27,889 INFO [Listener at localhost/34615] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44039 2023-07-12 19:04:27,889 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,890 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,891 INFO [Listener at localhost/34615] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44039 connecting to ZooKeeper ensemble=127.0.0.1:61051 2023-07-12 19:04:27,893 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:440390x0, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:27,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:44039-0x1015aec13380001 connected 2023-07-12 19:04:27,896 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:27,897 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:27,897 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:27,897 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44039 2023-07-12 19:04:27,898 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44039 2023-07-12 19:04:27,898 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44039 2023-07-12 19:04:27,898 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44039 2023-07-12 19:04:27,898 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44039 2023-07-12 19:04:27,898 INFO [Listener at localhost/34615] master.HMaster(444): hbase.rootdir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589, hbase.cluster.distributed=false 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:27,911 INFO [Listener at localhost/34615] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:27,912 INFO [Listener at localhost/34615] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39351 2023-07-12 19:04:27,912 INFO [Listener at localhost/34615] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:04:27,913 DEBUG [Listener at localhost/34615] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:04:27,914 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,915 INFO [Listener at localhost/34615] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:27,916 INFO [Listener at localhost/34615] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39351 connecting to ZooKeeper ensemble=127.0.0.1:61051 2023-07-12 19:04:27,919 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:393510x0, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:27,920 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): regionserver:393510x0, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:27,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39351-0x1015aec13380002 connected 2023-07-12 19:04:27,921 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:27,921 DEBUG [Listener at localhost/34615] zookeeper.ZKUtil(164): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:27,921 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39351 2023-07-12 19:04:27,922 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39351 2023-07-12 19:04:27,922 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39351 2023-07-12 19:04:27,922 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39351 2023-07-12 19:04:27,922 DEBUG [Listener at localhost/34615] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39351 2023-07-12 19:04:27,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:27,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:27,926 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:27,926 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:27,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:27,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:27,928 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:27,928 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:27,929 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:27,929 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:27,929 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:27,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,38203,1689188667859 from backup master directory 2023-07-12 19:04:27,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:27,930 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:27,930 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:27,931 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:27,931 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:27,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:27,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase4.apache.org,38203,1689188667859; waiting to become the next active master 2023-07-12 19:04:27,932 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:27,932 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:27,931 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:28,347 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/hbase.id with ID: ea6236b8-5c83-40f9-9673-106f1be53845 2023-07-12 19:04:28,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:28,361 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,361 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6abbb7eb to 127.0.0.1:61051 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:28,370 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5dd36577, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:28,370 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:28,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:28,371 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:28,372 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store-tmp 2023-07-12 19:04:28,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:28,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:28,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:28,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:28,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:28,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:28,780 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:28,780 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:28,781 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:28,783 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38203%2C1689188667859, suffix=, logDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859, archiveDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:28,797 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK] 2023-07-12 19:04:28,799 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:28,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK]] 2023-07-12 19:04:28,799 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:28,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:28,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,801 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,802 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:28,802 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:28,803 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:28,804 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,804 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,806 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:28,808 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:28,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66132515, jitterRate=-0.014548733830451965}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:28,809 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:28,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:28,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:28,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:28,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:28,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:28,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 19:04:28,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:28,811 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:04:28,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:28,822 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:28,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:28,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:28,823 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:28,823 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:28,825 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,825 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,826 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:28,826 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:28,827 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:28,829 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:28,829 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:28,829 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,829 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:28,829 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,830 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,38203,1689188667859, sessionid=0x1015aec13380000, setting cluster-up flag (Was=false) 2023-07-12 19:04:28,832 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegionServer(951): ClusterId : ea6236b8-5c83-40f9-9673-106f1be53845 2023-07-12 19:04:28,833 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,833 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:28,842 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:28,845 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,845 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:28,849 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:28,849 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:28,850 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.hbase-snapshot/.tmp 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:28,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188698852 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:28,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:28,854 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:28,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:28,855 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:28,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188668855,5,FailOnTimeoutGroup] 2023-07-12 19:04:28,855 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188668855,5,FailOnTimeoutGroup] 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,855 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,856 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:28,925 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(951): ClusterId : ea6236b8-5c83-40f9-9673-106f1be53845 2023-07-12 19:04:28,925 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:04:28,927 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:04:28,927 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:04:28,930 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:04:28,931 DEBUG [RS:0;jenkins-hbase4:39351] zookeeper.ReadOnlyZKClient(139): Connect 0x29e5211d to 127.0.0.1:61051 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:28,935 DEBUG [RS:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2722ca51, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:28,935 DEBUG [RS:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@231d4a6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:28,949 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:39351 2023-07-12 19:04:28,949 INFO [RS:0;jenkins-hbase4:39351] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:04:28,949 INFO [RS:0;jenkins-hbase4:39351] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:04:28,949 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:04:28,950 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,38203,1689188667859 with isa=jenkins-hbase4.apache.org/172.31.14.131:39351, startcode=1689188667910 2023-07-12 19:04:28,950 DEBUG [RS:0;jenkins-hbase4:39351] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:28,952 INFO [RS-EventLoopGroup-12-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41759, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:28,954 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38203] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,954 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589 2023-07-12 19:04:28,954 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:40837 2023-07-12 19:04:28,954 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:04:28,956 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:28,956 DEBUG [RS:0;jenkins-hbase4:39351] zookeeper.ZKUtil(162): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,956 WARN [RS:0;jenkins-hbase4:39351] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:28,956 INFO [RS:0;jenkins-hbase4:39351] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:28,956 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1948): logDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,957 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39351,1689188667910] 2023-07-12 19:04:28,960 DEBUG [RS:0;jenkins-hbase4:39351] zookeeper.ZKUtil(162): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,961 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:04:28,961 INFO [RS:0;jenkins-hbase4:39351] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:04:28,962 INFO [RS:0;jenkins-hbase4:39351] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:04:28,962 INFO [RS:0;jenkins-hbase4:39351] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:04:28,963 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,963 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:04:28,964 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 DEBUG [RS:0;jenkins-hbase4:39351] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:28,964 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,965 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,965 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,976 INFO [RS:0;jenkins-hbase4:39351] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:04:28,976 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39351,1689188667910-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:28,987 INFO [RS:0;jenkins-hbase4:39351] regionserver.Replication(203): jenkins-hbase4.apache.org,39351,1689188667910 started 2023-07-12 19:04:28,987 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39351,1689188667910, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39351, sessionid=0x1015aec13380002 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39351,1689188667910' 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:28,987 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39351,1689188667910' 2023-07-12 19:04:28,988 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:04:28,988 DEBUG [RS:0;jenkins-hbase4:39351] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:04:28,988 DEBUG [RS:0;jenkins-hbase4:39351] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:04:28,988 INFO [RS:0;jenkins-hbase4:39351] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 19:04:28,988 INFO [RS:0;jenkins-hbase4:39351] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 19:04:29,090 INFO [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39351%2C1689188667910, suffix=, logDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910, archiveDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/oldWALs, maxLogs=32 2023-07-12 19:04:29,105 DEBUG [RS-EventLoopGroup-14-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK] 2023-07-12 19:04:29,107 INFO [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910/jenkins-hbase4.apache.org%2C39351%2C1689188667910.1689188669091 2023-07-12 19:04:29,107 DEBUG [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK]] 2023-07-12 19:04:29,264 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:29,264 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:29,265 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589 2023-07-12 19:04:29,671 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:29,672 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:29,673 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/info 2023-07-12 19:04:29,674 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:29,675 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:29,675 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:29,676 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:29,676 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:29,677 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:29,677 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:29,678 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/table 2023-07-12 19:04:29,678 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:29,679 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:29,679 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740 2023-07-12 19:04:29,680 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740 2023-07-12 19:04:29,682 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:29,683 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:29,685 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66571497, jitterRate=-0.008007392287254333}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:29,685 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:29,685 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:29,686 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:29,686 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:29,687 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:29,687 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:04:29,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:04:29,689 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:04:29,690 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:04:29,840 DEBUG [jenkins-hbase4:38203] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:04:29,841 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39351,1689188667910, state=OPENING 2023-07-12 19:04:29,842 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:04:29,847 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:29,847 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:29,848 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:29,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:29,848 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:30,001 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:30,002 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:30,004 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48328, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:30,007 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:04:30,007 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:30,009 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39351%2C1689188667910.meta, suffix=.meta, logDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910, archiveDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/oldWALs, maxLogs=32 2023-07-12 19:04:30,024 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK] 2023-07-12 19:04:30,026 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910/jenkins-hbase4.apache.org%2C39351%2C1689188667910.meta.1689188670010.meta 2023-07-12 19:04:30,026 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK]] 2023-07-12 19:04:30,026 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:30,026 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:04:30,026 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:04:30,027 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:04:30,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:04:30,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:30,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:04:30,027 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:04:30,028 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:30,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/info 2023-07-12 19:04:30,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/info 2023-07-12 19:04:30,029 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:30,030 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:30,030 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:30,031 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:30,031 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:30,031 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:30,032 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:30,032 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:30,032 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/table 2023-07-12 19:04:30,033 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/table 2023-07-12 19:04:30,033 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:30,033 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:30,034 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740 2023-07-12 19:04:30,035 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740 2023-07-12 19:04:30,037 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:30,038 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:30,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66804585, jitterRate=-0.004534110426902771}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:30,039 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:30,039 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689188670001 2023-07-12 19:04:30,042 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:04:30,043 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:04:30,043 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39351,1689188667910, state=OPEN 2023-07-12 19:04:30,049 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:30,049 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:30,049 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:30,049 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:30,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:04:30,051 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39351,1689188667910 in 201 msec 2023-07-12 19:04:30,052 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:04:30,053 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 364 msec 2023-07-12 19:04:30,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.2020 sec 2023-07-12 19:04:30,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188670054, completionTime=-1 2023-07-12 19:04:30,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:30,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:30,057 DEBUG [hconnection-0x64db4b27-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:30,059 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48338, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:30,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:30,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188730060 2023-07-12 19:04:30,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188790060 2023-07-12 19:04:30,060 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38203,1689188667859-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38203,1689188667859-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38203,1689188667859-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:38203, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:30,065 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:04:30,066 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:30,066 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:30,066 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:30,068 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:30,068 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:30,070 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:30,070 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb empty. 2023-07-12 19:04:30,070 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:30,070 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:04:30,479 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:30,480 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4e5be26357fae50a1ff6b45ead0d6fbb, NAME => 'hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:30,742 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:30,886 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:30,887 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 4e5be26357fae50a1ff6b45ead0d6fbb, disabling compactions & flushes 2023-07-12 19:04:30,887 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:30,887 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:30,887 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. after waiting 0 ms 2023-07-12 19:04:30,887 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:30,887 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:30,887 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 4e5be26357fae50a1ff6b45ead0d6fbb: 2023-07-12 19:04:30,889 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:30,890 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188670890"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188670890"}]},"ts":"1689188670890"} 2023-07-12 19:04:30,893 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:30,894 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:30,894 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188670894"}]},"ts":"1689188670894"} 2023-07-12 19:04:30,895 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:04:30,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4e5be26357fae50a1ff6b45ead0d6fbb, ASSIGN}] 2023-07-12 19:04:30,901 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=4e5be26357fae50a1ff6b45ead0d6fbb, ASSIGN 2023-07-12 19:04:30,902 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=4e5be26357fae50a1ff6b45ead0d6fbb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:31,053 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:31,053 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188671053"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188671053"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188671053"}]},"ts":"1689188671053"} 2023-07-12 19:04:31,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 4e5be26357fae50a1ff6b45ead0d6fbb, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:31,211 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:31,211 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4e5be26357fae50a1ff6b45ead0d6fbb, NAME => 'hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:31,211 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,212 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:31,212 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,212 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,213 INFO [StoreOpener-4e5be26357fae50a1ff6b45ead0d6fbb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,214 DEBUG [StoreOpener-4e5be26357fae50a1ff6b45ead0d6fbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/info 2023-07-12 19:04:31,214 DEBUG [StoreOpener-4e5be26357fae50a1ff6b45ead0d6fbb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/info 2023-07-12 19:04:31,215 INFO [StoreOpener-4e5be26357fae50a1ff6b45ead0d6fbb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e5be26357fae50a1ff6b45ead0d6fbb columnFamilyName info 2023-07-12 19:04:31,215 INFO [StoreOpener-4e5be26357fae50a1ff6b45ead0d6fbb-1] regionserver.HStore(310): Store=4e5be26357fae50a1ff6b45ead0d6fbb/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:31,216 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,216 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,219 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:31,220 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:31,221 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4e5be26357fae50a1ff6b45ead0d6fbb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72616588, jitterRate=0.08207148313522339}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:31,221 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4e5be26357fae50a1ff6b45ead0d6fbb: 2023-07-12 19:04:31,222 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb., pid=6, masterSystemTime=1689188671207 2023-07-12 19:04:31,224 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:31,224 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:31,224 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:31,225 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188671224"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188671224"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188671224"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188671224"}]},"ts":"1689188671224"} 2023-07-12 19:04:31,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:04:31,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 4e5be26357fae50a1ff6b45ead0d6fbb, server=jenkins-hbase4.apache.org,39351,1689188667910 in 171 msec 2023-07-12 19:04:31,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:04:31,229 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=4e5be26357fae50a1ff6b45ead0d6fbb, ASSIGN in 329 msec 2023-07-12 19:04:31,230 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:31,230 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188671230"}]},"ts":"1689188671230"} 2023-07-12 19:04:31,232 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:04:31,234 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:31,235 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1680 sec 2023-07-12 19:04:31,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:04:31,269 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:31,269 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:31,269 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:31,274 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:31,283 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:31,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 12 msec 2023-07-12 19:04:31,296 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:31,303 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:31,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 9 msec 2023-07-12 19:04:31,310 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:31,312 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:31,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.380sec 2023-07-12 19:04:31,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:31,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:31,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:31,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38203,1689188667859-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:31,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38203,1689188667859-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:31,315 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:31,327 DEBUG [Listener at localhost/34615] zookeeper.ReadOnlyZKClient(139): Connect 0x1088e925 to 127.0.0.1:61051 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:31,332 DEBUG [Listener at localhost/34615] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7fafcf28, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:31,333 DEBUG [hconnection-0x4e34f32d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:31,335 INFO [RS-EventLoopGroup-14-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:31,337 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:31,337 WARN [Listener at localhost/34615] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:31,340 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:31,341 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:31,342 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:31,346 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:31,346 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:31,346 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:31,346 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 empty. 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 empty. 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 empty. 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 empty. 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:31,347 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:31,348 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:31,348 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:31,348 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 19:04:31,756 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:31,757 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5bb0a9498878cf2bd91ed66e846a7170, NAME => 'testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:31,758 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 79113b41be6b3e023e187f2b1fd17460, NAME => 'testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:31,758 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => fc6cf34c3ef9229ba8327618b2701f07, NAME => 'testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:31,766 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:31,767 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing fc6cf34c3ef9229ba8327618b2701f07, disabling compactions & flushes 2023-07-12 19:04:31,767 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:31,767 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:31,767 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. after waiting 0 ms 2023-07-12 19:04:31,767 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:31,767 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:31,767 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for fc6cf34c3ef9229ba8327618b2701f07: 2023-07-12 19:04:31,767 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 1a12a39783f6f1f6efd1a03e6dd6a9a3, NAME => 'testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:31,772 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:31,772 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 1a12a39783f6f1f6efd1a03e6dd6a9a3, disabling compactions & flushes 2023-07-12 19:04:31,772 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:31,772 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:31,772 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. after waiting 0 ms 2023-07-12 19:04:31,773 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:31,773 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:31,773 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 1a12a39783f6f1f6efd1a03e6dd6a9a3: 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 5bb0a9498878cf2bd91ed66e846a7170, disabling compactions & flushes 2023-07-12 19:04:32,167 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. after waiting 0 ms 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,167 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 5bb0a9498878cf2bd91ed66e846a7170: 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 79113b41be6b3e023e187f2b1fd17460, disabling compactions & flushes 2023-07-12 19:04:32,167 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,167 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,168 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. after waiting 0 ms 2023-07-12 19:04:32,168 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,168 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,168 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 79113b41be6b3e023e187f2b1fd17460: 2023-07-12 19:04:32,170 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:32,171 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188672171"}]},"ts":"1689188672171"} 2023-07-12 19:04:32,171 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188672171"}]},"ts":"1689188672171"} 2023-07-12 19:04:32,171 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188672171"}]},"ts":"1689188672171"} 2023-07-12 19:04:32,171 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672171"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188672171"}]},"ts":"1689188672171"} 2023-07-12 19:04:32,175 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 19:04:32,176 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:32,176 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188672176"}]},"ts":"1689188672176"} 2023-07-12 19:04:32,178 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 19:04:32,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, ASSIGN}] 2023-07-12 19:04:32,183 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, ASSIGN 2023-07-12 19:04:32,184 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:32,185 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, ASSIGN 2023-07-12 19:04:32,185 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:32,186 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, ASSIGN 2023-07-12 19:04:32,187 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:32,187 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, ASSIGN 2023-07-12 19:04:32,188 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:32,336 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,337 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672336"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672336"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672336"}]},"ts":"1689188672336"} 2023-07-12 19:04:32,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; OpenRegionProcedure 1a12a39783f6f1f6efd1a03e6dd6a9a3, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,342 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5bb0a9498878cf2bd91ed66e846a7170, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,342 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672342"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672342"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672342"}]},"ts":"1689188672342"} 2023-07-12 19:04:32,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=10, state=RUNNABLE; OpenRegionProcedure 5bb0a9498878cf2bd91ed66e846a7170, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,346 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=fc6cf34c3ef9229ba8327618b2701f07, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,346 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672346"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672346"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672346"}]},"ts":"1689188672346"} 2023-07-12 19:04:32,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=11, state=RUNNABLE; OpenRegionProcedure fc6cf34c3ef9229ba8327618b2701f07, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,350 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=79113b41be6b3e023e187f2b1fd17460, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,350 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672350"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672350"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672350"}]},"ts":"1689188672350"} 2023-07-12 19:04:32,351 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE; OpenRegionProcedure 79113b41be6b3e023e187f2b1fd17460, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:32,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc6cf34c3ef9229ba8327618b2701f07, NAME => 'testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 19:04:32,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,496 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,497 DEBUG [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1 2023-07-12 19:04:32,498 DEBUG [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1 2023-07-12 19:04:32,498 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc6cf34c3ef9229ba8327618b2701f07 columnFamilyName f1 2023-07-12 19:04:32,498 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] regionserver.HStore(310): Store=fc6cf34c3ef9229ba8327618b2701f07/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,499 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,500 DEBUG [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2 2023-07-12 19:04:32,500 DEBUG [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2 2023-07-12 19:04:32,500 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc6cf34c3ef9229ba8327618b2701f07 columnFamilyName f2 2023-07-12 19:04:32,501 INFO [StoreOpener-fc6cf34c3ef9229ba8327618b2701f07-1] regionserver.HStore(310): Store=fc6cf34c3ef9229ba8327618b2701f07/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,502 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:32,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:32,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:32,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fc6cf34c3ef9229ba8327618b2701f07; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74712485, jitterRate=0.11330278217792511}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:32,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fc6cf34c3ef9229ba8327618b2701f07: 2023-07-12 19:04:32,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07., pid=16, masterSystemTime=1689188672491 2023-07-12 19:04:32,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:32,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:32,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1a12a39783f6f1f6efd1a03e6dd6a9a3, NAME => 'testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 19:04:32,510 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=fc6cf34c3ef9229ba8327618b2701f07, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,510 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672509"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188672509"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188672509"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188672509"}]},"ts":"1689188672509"} 2023-07-12 19:04:32,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,510 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,511 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,513 DEBUG [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1 2023-07-12 19:04:32,513 DEBUG [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1 2023-07-12 19:04:32,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=11 2023-07-12 19:04:32,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=11, state=SUCCESS; OpenRegionProcedure fc6cf34c3ef9229ba8327618b2701f07, server=jenkins-hbase4.apache.org,39351,1689188667910 in 163 msec 2023-07-12 19:04:32,513 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a12a39783f6f1f6efd1a03e6dd6a9a3 columnFamilyName f1 2023-07-12 19:04:32,514 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] regionserver.HStore(310): Store=1a12a39783f6f1f6efd1a03e6dd6a9a3/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,514 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, ASSIGN in 332 msec 2023-07-12 19:04:32,515 DEBUG [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2 2023-07-12 19:04:32,515 DEBUG [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2 2023-07-12 19:04:32,515 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1a12a39783f6f1f6efd1a03e6dd6a9a3 columnFamilyName f2 2023-07-12 19:04:32,516 INFO [StoreOpener-1a12a39783f6f1f6efd1a03e6dd6a9a3-1] regionserver.HStore(310): Store=1a12a39783f6f1f6efd1a03e6dd6a9a3/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,519 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:32,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:32,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1a12a39783f6f1f6efd1a03e6dd6a9a3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67244096, jitterRate=0.0020151138305664062}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:32,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1a12a39783f6f1f6efd1a03e6dd6a9a3: 2023-07-12 19:04:32,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3., pid=14, masterSystemTime=1689188672491 2023-07-12 19:04:32,524 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,525 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5bb0a9498878cf2bd91ed66e846a7170, NAME => 'testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 19:04:32,525 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,525 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672525"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188672525"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188672525"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188672525"}]},"ts":"1689188672525"} 2023-07-12 19:04:32,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,525 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,526 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,528 DEBUG [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1 2023-07-12 19:04:32,528 DEBUG [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1 2023-07-12 19:04:32,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-12 19:04:32,528 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; OpenRegionProcedure 1a12a39783f6f1f6efd1a03e6dd6a9a3, server=jenkins-hbase4.apache.org,39351,1689188667910 in 187 msec 2023-07-12 19:04:32,528 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5bb0a9498878cf2bd91ed66e846a7170 columnFamilyName f1 2023-07-12 19:04:32,529 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] regionserver.HStore(310): Store=5bb0a9498878cf2bd91ed66e846a7170/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,529 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,530 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, ASSIGN in 347 msec 2023-07-12 19:04:32,530 DEBUG [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2 2023-07-12 19:04:32,530 DEBUG [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2 2023-07-12 19:04:32,531 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5bb0a9498878cf2bd91ed66e846a7170 columnFamilyName f2 2023-07-12 19:04:32,531 INFO [StoreOpener-5bb0a9498878cf2bd91ed66e846a7170-1] regionserver.HStore(310): Store=5bb0a9498878cf2bd91ed66e846a7170/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,532 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:32,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:32,536 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:32,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5bb0a9498878cf2bd91ed66e846a7170; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63419418, jitterRate=-0.05497702956199646}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:32,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5bb0a9498878cf2bd91ed66e846a7170: 2023-07-12 19:04:32,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170., pid=15, masterSystemTime=1689188672491 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:32,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 79113b41be6b3e023e187f2b1fd17460, NAME => 'testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 19:04:32,539 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5bb0a9498878cf2bd91ed66e846a7170, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:32,539 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672539"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188672539"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188672539"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188672539"}]},"ts":"1689188672539"} 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,539 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,540 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,542 DEBUG [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1 2023-07-12 19:04:32,542 DEBUG [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1 2023-07-12 19:04:32,542 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 79113b41be6b3e023e187f2b1fd17460 columnFamilyName f1 2023-07-12 19:04:32,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=10 2023-07-12 19:04:32,542 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=10, state=SUCCESS; OpenRegionProcedure 5bb0a9498878cf2bd91ed66e846a7170, server=jenkins-hbase4.apache.org,39351,1689188667910 in 197 msec 2023-07-12 19:04:32,543 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] regionserver.HStore(310): Store=79113b41be6b3e023e187f2b1fd17460/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,543 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,543 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, ASSIGN in 361 msec 2023-07-12 19:04:32,544 DEBUG [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2 2023-07-12 19:04:32,544 DEBUG [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2 2023-07-12 19:04:32,544 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 79113b41be6b3e023e187f2b1fd17460 columnFamilyName f2 2023-07-12 19:04:32,545 INFO [StoreOpener-79113b41be6b3e023e187f2b1fd17460-1] regionserver.HStore(310): Store=79113b41be6b3e023e187f2b1fd17460/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:32,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,547 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:32,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:32,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 79113b41be6b3e023e187f2b1fd17460; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73133418, jitterRate=0.08977285027503967}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:32,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 79113b41be6b3e023e187f2b1fd17460: 2023-07-12 19:04:32,551 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460., pid=17, masterSystemTime=1689188672491 2023-07-12 19:04:32,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:32,552 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=79113b41be6b3e023e187f2b1fd17460, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,553 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672552"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188672552"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188672552"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188672552"}]},"ts":"1689188672552"} 2023-07-12 19:04:32,555 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-12 19:04:32,556 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; OpenRegionProcedure 79113b41be6b3e023e187f2b1fd17460, server=jenkins-hbase4.apache.org,39351,1689188667910 in 203 msec 2023-07-12 19:04:32,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-07-12 19:04:32,557 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, ASSIGN in 375 msec 2023-07-12 19:04:32,558 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:32,558 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188672558"}]},"ts":"1689188672558"} 2023-07-12 19:04:32,559 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 19:04:32,561 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testTruncateWithFailoverAtStep4 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:32,562 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 in 1.2220 sec 2023-07-12 19:04:32,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:32,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:32,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:32,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:32,629 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 19:04:32,630 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 19:04:32,632 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 19:04:32,634 DEBUG [Listener at localhost/34615] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 19:04:32,636 INFO [RS-EventLoopGroup-12-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 19:04:32,636 INFO [Listener at localhost/34615] client.HBaseAdmin$15(890): Started disable of testTruncateWithFailoverAtStep4 2023-07-12 19:04:32,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testTruncateWithFailoverAtStep4 2023-07-12 19:04:32,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:32,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:32,641 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188672640"}]},"ts":"1689188672640"} 2023-07-12 19:04:32,642 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLING in hbase:meta 2023-07-12 19:04:32,644 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testTruncateWithFailoverAtStep4 to state=DISABLING 2023-07-12 19:04:32,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, UNASSIGN}] 2023-07-12 19:04:32,646 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, UNASSIGN 2023-07-12 19:04:32,646 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,646 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672646"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672646"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672646"}]},"ts":"1689188672646"} 2023-07-12 19:04:32,648 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure 1a12a39783f6f1f6efd1a03e6dd6a9a3, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,650 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, UNASSIGN 2023-07-12 19:04:32,651 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=79113b41be6b3e023e187f2b1fd17460, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,651 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672650"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672650"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672650"}]},"ts":"1689188672650"} 2023-07-12 19:04:32,652 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure 79113b41be6b3e023e187f2b1fd17460, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,654 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, UNASSIGN 2023-07-12 19:04:32,655 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=fc6cf34c3ef9229ba8327618b2701f07, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,655 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188672655"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672655"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672655"}]},"ts":"1689188672655"} 2023-07-12 19:04:32,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure fc6cf34c3ef9229ba8327618b2701f07, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,659 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, UNASSIGN 2023-07-12 19:04:32,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=5bb0a9498878cf2bd91ed66e846a7170, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:32,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188672659"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188672659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188672659"}]},"ts":"1689188672659"} 2023-07-12 19:04:32,661 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 5bb0a9498878cf2bd91ed66e846a7170, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:32,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:32,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:32,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1a12a39783f6f1f6efd1a03e6dd6a9a3, disabling compactions & flushes 2023-07-12 19:04:32,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. after waiting 0 ms 2023-07-12 19:04:32,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:32,801 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1a12a39783f6f1f6efd1a03e6dd6a9a3 2/2 column families, dataSize=4.05 KB heapSize=9.22 KB 2023-07-12 19:04:32,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:33,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.03 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/.tmp/f1/0adf450c4e7542b28672cd362827e0d2 2023-07-12 19:04:33,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.03 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/.tmp/f2/b4a695b232874ec09d37ab09cd1bece7 2023-07-12 19:04:33,231 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/.tmp/f1/0adf450c4e7542b28672cd362827e0d2 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1/0adf450c4e7542b28672cd362827e0d2 2023-07-12 19:04:33,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1/0adf450c4e7542b28672cd362827e0d2, entries=31, sequenceid=5, filesize=6.9 K 2023-07-12 19:04:33,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/.tmp/f2/b4a695b232874ec09d37ab09cd1bece7 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2/b4a695b232874ec09d37ab09cd1bece7 2023-07-12 19:04:33,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2/b4a695b232874ec09d37ab09cd1bece7, entries=31, sequenceid=5, filesize=6.9 K 2023-07-12 19:04:33,243 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.05 KB/4150, heapSize ~9.19 KB/9408, currentSize=0 B/0 for 1a12a39783f6f1f6efd1a03e6dd6a9a3 in 442ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:33,243 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-12 19:04:33,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:33,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:33,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. 2023-07-12 19:04:33,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1a12a39783f6f1f6efd1a03e6dd6a9a3: 2023-07-12 19:04:33,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:33,252 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:33,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5bb0a9498878cf2bd91ed66e846a7170, disabling compactions & flushes 2023-07-12 19:04:33,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:33,253 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:33,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. after waiting 0 ms 2023-07-12 19:04:33,254 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:33,254 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5bb0a9498878cf2bd91ed66e846a7170 2/2 column families, dataSize=7.33 KB heapSize=16.25 KB 2023-07-12 19:04:33,254 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=CLOSED 2023-07-12 19:04:33,254 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188673254"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188673254"}]},"ts":"1689188673254"} 2023-07-12 19:04:33,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 19:04:33,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure 1a12a39783f6f1f6efd1a03e6dd6a9a3, server=jenkins-hbase4.apache.org,39351,1689188667910 in 608 msec 2023-07-12 19:04:33,259 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=1a12a39783f6f1f6efd1a03e6dd6a9a3, UNASSIGN in 612 msec 2023-07-12 19:04:33,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.66 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/.tmp/f1/60099ef9d1894c028d8854d00ac3542e 2023-07-12 19:04:33,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:34,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.66 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/.tmp/f2/a86ce4efcbfa40df9bb485399278c82f 2023-07-12 19:04:34,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/.tmp/f1/60099ef9d1894c028d8854d00ac3542e as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1/60099ef9d1894c028d8854d00ac3542e 2023-07-12 19:04:34,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1/60099ef9d1894c028d8854d00ac3542e, entries=56, sequenceid=5, filesize=8.7 K 2023-07-12 19:04:34,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/.tmp/f2/a86ce4efcbfa40df9bb485399278c82f as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2/a86ce4efcbfa40df9bb485399278c82f 2023-07-12 19:04:34,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2/a86ce4efcbfa40df9bb485399278c82f, entries=56, sequenceid=5, filesize=8.7 K 2023-07-12 19:04:34,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~7.33 KB/7504, heapSize ~16.22 KB/16608, currentSize=0 B/0 for 5bb0a9498878cf2bd91ed66e846a7170 in 842ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:34,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:34,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. 2023-07-12 19:04:34,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5bb0a9498878cf2bd91ed66e846a7170: 2023-07-12 19:04:34,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:34,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:34,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fc6cf34c3ef9229ba8327618b2701f07, disabling compactions & flushes 2023-07-12 19:04:34,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:34,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:34,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. after waiting 0 ms 2023-07-12 19:04:34,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:34,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fc6cf34c3ef9229ba8327618b2701f07 2/2 column families, dataSize=666 B heapSize=1.91 KB 2023-07-12 19:04:34,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=5bb0a9498878cf2bd91ed66e846a7170, regionState=CLOSED 2023-07-12 19:04:34,107 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188674107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188674107"}]},"ts":"1689188674107"} 2023-07-12 19:04:34,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 19:04:34,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 5bb0a9498878cf2bd91ed66e846a7170, server=jenkins-hbase4.apache.org,39351,1689188667910 in 1.4480 sec 2023-07-12 19:04:34,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5bb0a9498878cf2bd91ed66e846a7170, UNASSIGN in 1.4660 sec 2023-07-12 19:04:34,516 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=333 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/.tmp/f1/477c115b997d4dfb98fd35b57fb1f992 2023-07-12 19:04:34,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:34,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=333 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/.tmp/f2/0e3ae7eb375f4b8b8d35eaccb3288d27 2023-07-12 19:04:34,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/.tmp/f1/477c115b997d4dfb98fd35b57fb1f992 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1/477c115b997d4dfb98fd35b57fb1f992 2023-07-12 19:04:34,939 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1/477c115b997d4dfb98fd35b57fb1f992, entries=5, sequenceid=5, filesize=5.2 K 2023-07-12 19:04:34,940 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/.tmp/f2/0e3ae7eb375f4b8b8d35eaccb3288d27 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2/0e3ae7eb375f4b8b8d35eaccb3288d27 2023-07-12 19:04:34,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2/0e3ae7eb375f4b8b8d35eaccb3288d27, entries=5, sequenceid=5, filesize=5.2 K 2023-07-12 19:04:34,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~666 B/666, heapSize ~1.88 KB/1920, currentSize=0 B/0 for fc6cf34c3ef9229ba8327618b2701f07 in 839ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:34,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:34,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. 2023-07-12 19:04:34,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fc6cf34c3ef9229ba8327618b2701f07: 2023-07-12 19:04:34,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:34,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:34,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 79113b41be6b3e023e187f2b1fd17460, disabling compactions & flushes 2023-07-12 19:04:34,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:34,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:34,954 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=fc6cf34c3ef9229ba8327618b2701f07, regionState=CLOSED 2023-07-12 19:04:34,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. after waiting 0 ms 2023-07-12 19:04:34,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:34,954 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188674954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188674954"}]},"ts":"1689188674954"} 2023-07-12 19:04:34,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 79113b41be6b3e023e187f2b1fd17460 2/2 column families, dataSize=1.04 KB heapSize=2.75 KB 2023-07-12 19:04:34,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 19:04:34,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure fc6cf34c3ef9229ba8327618b2701f07, server=jenkins-hbase4.apache.org,39351,1689188667910 in 2.2990 sec 2023-07-12 19:04:34,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=fc6cf34c3ef9229ba8327618b2701f07, UNASSIGN in 2.3120 sec 2023-07-12 19:04:34,961 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 19:04:34,962 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 19:04:34,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=534 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/.tmp/f1/895de7ee03874b059874a04ed6dd9184 2023-07-12 19:04:35,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=534 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/.tmp/f2/6ae25162f9cf49fa96d8bd6cab32a737 2023-07-12 19:04:35,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/.tmp/f1/895de7ee03874b059874a04ed6dd9184 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1/895de7ee03874b059874a04ed6dd9184 2023-07-12 19:04:35,385 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1/895de7ee03874b059874a04ed6dd9184, entries=8, sequenceid=5, filesize=5.4 K 2023-07-12 19:04:35,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/.tmp/f2/6ae25162f9cf49fa96d8bd6cab32a737 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2/6ae25162f9cf49fa96d8bd6cab32a737 2023-07-12 19:04:35,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2/6ae25162f9cf49fa96d8bd6cab32a737, entries=8, sequenceid=5, filesize=5.4 K 2023-07-12 19:04:35,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.04 KB/1068, heapSize ~2.72 KB/2784, currentSize=0 B/0 for 79113b41be6b3e023e187f2b1fd17460 in 437ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:35,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:35,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. 2023-07-12 19:04:35,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 79113b41be6b3e023e187f2b1fd17460: 2023-07-12 19:04:35,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:35,400 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=79113b41be6b3e023e187f2b1fd17460, regionState=CLOSED 2023-07-12 19:04:35,400 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188675400"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188675400"}]},"ts":"1689188675400"} 2023-07-12 19:04:35,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 19:04:35,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure 79113b41be6b3e023e187f2b1fd17460, server=jenkins-hbase4.apache.org,39351,1689188667910 in 2.7490 sec 2023-07-12 19:04:35,404 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-07-12 19:04:35,404 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=79113b41be6b3e023e187f2b1fd17460, UNASSIGN in 2.7580 sec 2023-07-12 19:04:35,405 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188675404"}]},"ts":"1689188675404"} 2023-07-12 19:04:35,406 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=DISABLED in hbase:meta 2023-07-12 19:04:35,407 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testTruncateWithFailoverAtStep4 to state=DISABLED 2023-07-12 19:04:35,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 in 2.7710 sec 2023-07-12 19:04:36,714 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:36,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38203] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:36,753 INFO [Listener at localhost/34615] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testTruncateWithFailoverAtStep4, procId: 18 completed 2023-07-12 19:04:36,753 WARN [Listener at localhost/34615] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 19:04:36,759 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:36,761 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-12 19:04:36,761 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:36,761 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:36,761 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:36,761 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:37,009 INFO [Listener at localhost/34615] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,010 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:37,010 INFO [Listener at localhost/34615] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:37,010 INFO [Listener at localhost/34615] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:37,012 ERROR [Listener at localhost/34615] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:37,012 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:37,013 INFO [Listener at localhost/34615] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:37,013 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:37,013 INFO [Listener at localhost/34615] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:37,013 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:37,014 DEBUG [Listener at localhost/34615] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[] 2023-07-12 19:04:37,015 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:37,015 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:37,015 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:37,015 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:37,016 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,016 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,016 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,016 DEBUG [Listener at localhost/34615] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:37,017 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:37,017 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:37,018 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39351,1689188667910, table=hbase:meta, region=1588230740 2023-07-12 19:04:37,018 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188707018 2023-07-12 19:04:37,019 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_PRE_OPERATION; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,019 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:37,021 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,022 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=5bb0a9498878cf2bd91ed66e846a7170, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,022 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=fc6cf34c3ef9229ba8327618b2701f07, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,022 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=79113b41be6b3e023e187f2b1fd17460, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,022 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,022 INFO [Listener at localhost/34615] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:37,022 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188737022 2023-07-12 19:04:37,022 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188797022 2023-07-12 19:04:37,022 INFO [Listener at localhost/34615] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:37,022 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(87): waiting for 'testTruncateWithFailoverAtStep4' regions in transition 2023-07-12 19:04:37,023 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:37,030 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:37,030 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:37,030 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:37,030 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:37,033 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/recovered.edits] 2023-07-12 19:04:37,033 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/recovered.edits] 2023-07-12 19:04:37,033 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/recovered.edits] 2023-07-12 19:04:37,034 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2, FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/recovered.edits] 2023-07-12 19:04:37,039 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1/895de7ee03874b059874a04ed6dd9184 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f1/895de7ee03874b059874a04ed6dd9184 2023-07-12 19:04:37,039 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1/60099ef9d1894c028d8854d00ac3542e to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f1/60099ef9d1894c028d8854d00ac3542e 2023-07-12 19:04:37,039 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1/0adf450c4e7542b28672cd362827e0d2 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f1/0adf450c4e7542b28672cd362827e0d2 2023-07-12 19:04:37,040 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1/477c115b997d4dfb98fd35b57fb1f992 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f1/477c115b997d4dfb98fd35b57fb1f992 2023-07-12 19:04:37,044 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2/6ae25162f9cf49fa96d8bd6cab32a737 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/f2/6ae25162f9cf49fa96d8bd6cab32a737 2023-07-12 19:04:37,044 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2/b4a695b232874ec09d37ab09cd1bece7 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/f2/b4a695b232874ec09d37ab09cd1bece7 2023-07-12 19:04:37,044 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2/a86ce4efcbfa40df9bb485399278c82f to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/f2/a86ce4efcbfa40df9bb485399278c82f 2023-07-12 19:04:37,045 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2/0e3ae7eb375f4b8b8d35eaccb3288d27 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/f2/0e3ae7eb375f4b8b8d35eaccb3288d27 2023-07-12 19:04:37,049 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/recovered.edits/8.seqid to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460/recovered.edits/8.seqid 2023-07-12 19:04:37,049 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/recovered.edits/8.seqid to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3/recovered.edits/8.seqid 2023-07-12 19:04:37,049 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:37,050 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:37,051 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/recovered.edits/8.seqid to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170/recovered.edits/8.seqid 2023-07-12 19:04:37,052 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:37,052 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/recovered.edits/8.seqid to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/archive/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07/recovered.edits/8.seqid 2023-07-12 19:04:37,053 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:37,053 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 19:04:37,055 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:37,055 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,055 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:37,055 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:37,272 INFO [Listener at localhost/34615] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:37,273 ERROR [Listener at localhost/34615] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:37,273 INFO [Listener at localhost/34615] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:37,274 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:37,275 DEBUG [Listener at localhost/34615] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0] 2023-07-12 19:04:37,275 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:37,276 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:37,276 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:37,276 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:37,276 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,276 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,277 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,277 DEBUG [Listener at localhost/34615] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:37,277 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:37,278 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:37,278 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39351,1689188667910, table=hbase:meta, region=1588230740 2023-07-12 19:04:37,278 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188707278 2023-07-12 19:04:37,279 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:37,279 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,282 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,282 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=5bb0a9498878cf2bd91ed66e846a7170, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,282 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=fc6cf34c3ef9229ba8327618b2701f07, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,282 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=79113b41be6b3e023e187f2b1fd17460, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,283 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=1a12a39783f6f1f6efd1a03e6dd6a9a3, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,283 INFO [Listener at localhost/34615] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:37,283 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188737283 2023-07-12 19:04:37,283 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188797283 2023-07-12 19:04:37,283 INFO [Listener at localhost/34615] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:37,287 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:37,287 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:37,287 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:37,287 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 empty. 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 empty. 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 empty. 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 empty. 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5bb0a9498878cf2bd91ed66e846a7170 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/1a12a39783f6f1f6efd1a03e6dd6a9a3 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/79113b41be6b3e023e187f2b1fd17460 2023-07-12 19:04:37,288 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/fc6cf34c3ef9229ba8327618b2701f07 2023-07-12 19:04:37,289 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 19:04:37,290 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:37,294 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 4 rows of testTruncateWithFailoverAtStep4 from hbase:meta 2023-07-12 19:04:37,296 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-12 19:04:37,297 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-12 19:04:37,297 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689188677297"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,297 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689188677297"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,297 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689188677297"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,297 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1689188677297"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,299 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 4 regions from META 2023-07-12 19:04:37,299 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5bb0a9498878cf2bd91ed66e846a7170, NAME => 'testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170.', STARTKEY => '', ENDKEY => 'a'}, {ENCODED => fc6cf34c3ef9229ba8327618b2701f07, NAME => 'testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07.', STARTKEY => 'a', ENDKEY => 'b'}, {ENCODED => 79113b41be6b3e023e187f2b1fd17460, NAME => 'testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460.', STARTKEY => 'b', ENDKEY => 'c'}, {ENCODED => 1a12a39783f6f1f6efd1a03e6dd6a9a3, NAME => 'testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3.', STARTKEY => 'c', ENDKEY => ''}] 2023-07-12 19:04:37,299 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-12 19:04:37,299 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689188677299"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,301 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-12 19:04:37,302 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:37,302 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,302 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:37,302 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:37,533 INFO [Listener at localhost/34615] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,533 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:37,533 INFO [Listener at localhost/34615] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:37,533 INFO [Listener at localhost/34615] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:37,534 ERROR [Listener at localhost/34615] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:37,534 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:37,534 INFO [Listener at localhost/34615] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:37,534 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:37,534 INFO [Listener at localhost/34615] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:37,535 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:37,536 DEBUG [Listener at localhost/34615] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1] 2023-07-12 19:04:37,537 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:37,537 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:37,537 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:37,537 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:37,538 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,538 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:37,538 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,538 DEBUG [Listener at localhost/34615] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:37,538 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 19:04:37,539 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:37,539 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39351,1689188667910, table=hbase:meta, region=1588230740 2023-07-12 19:04:37,540 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188707540 2023-07-12 19:04:37,540 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:37,541 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_REMOVE_FROM_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:37,543 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:37,543 INFO [Listener at localhost/34615] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:37,543 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188737543 2023-07-12 19:04:37,543 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188797543 2023-07-12 19:04:37,543 INFO [Listener at localhost/34615] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:37,546 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testTruncateWithFailoverAtStep4' descriptor. 2023-07-12 19:04:37,546 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testTruncateWithFailoverAtStep4' from region states. 2023-07-12 19:04:37,546 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 0 regions from META 2023-07-12 19:04:37,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [] 2023-07-12 19:04:37,546 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testTruncateWithFailoverAtStep4' as deleted. 2023-07-12 19:04:37,546 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1689188677546"}]},"ts":"9223372036854775807"} 2023-07-12 19:04:37,548 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testTruncateWithFailoverAtStep4 state from META 2023-07-12 19:04:37,550 DEBUG [PEWorker-1] zookeeper.RecoverableZooKeeper(172): Node /hbase/table/testTruncateWithFailoverAtStep4 already deleted, retry=false 2023-07-12 19:04:37,550 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:37,557 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:37,557 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:37,557 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:37,557 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e empty. 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 empty. 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 empty. 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 empty. 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:37,558 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:37,559 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:37,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 19:04:37,971 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:37,973 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 5d792f1fb61887b34b7931e503a40374, NAME => 'testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:37,973 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => f78c8d9fe947191cb6c8b3d8acef835e, NAME => 'testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:37,973 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 602be7fec076ee1be772900bc4be2980, NAME => 'testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 5d792f1fb61887b34b7931e503a40374, disabling compactions & flushes 2023-07-12 19:04:38,381 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 602be7fec076ee1be772900bc4be2980, disabling compactions & flushes 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing f78c8d9fe947191cb6c8b3d8acef835e, disabling compactions & flushes 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:38,381 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. after waiting 0 ms 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. after waiting 0 ms 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for f78c8d9fe947191cb6c8b3d8acef835e: 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. after waiting 0 ms 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 5d792f1fb61887b34b7931e503a40374: 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:38,382 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 602be7fec076ee1be772900bc4be2980: 2023-07-12 19:04:38,382 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5c1f0e8b4afac0d7d73b1239993f8752, NAME => 'testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing 5c1f0e8b4afac0d7d73b1239993f8752, disabling compactions & flushes 2023-07-12 19:04:38,789 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. after waiting 0 ms 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:38,789 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:38,789 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for 5c1f0e8b4afac0d7d73b1239993f8752: 2023-07-12 19:04:38,791 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:38,791 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:38,791 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:38,791 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:38,794 INFO [Listener at localhost/34615] procedure.TestMasterFailoverWithProcedures(306): Restart 3 exec state: pid=27, state=RUNNABLE:TRUNCATE_TABLE_CLEAR_FS_LAYOUT, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:38,794 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:38,794 INFO [Listener at localhost/34615] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:38,794 INFO [Listener at localhost/34615] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:38,795 ERROR [Listener at localhost/34615] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:38,795 INFO [Listener at localhost/34615] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:38,795 INFO [Listener at localhost/34615] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:38,795 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:38,795 INFO [Listener at localhost/34615] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:38,795 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:38,796 DEBUG [Listener at localhost/34615] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2] 2023-07-12 19:04:38,797 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:38,797 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:38,797 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:38,797 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:38,798 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:38,798 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:38,798 DEBUG [Listener at localhost/34615] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:38,798 DEBUG [Listener at localhost/34615] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:38,798 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:38,799 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:38,799 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39351,1689188667910, table=hbase:meta, region=1588230740 2023-07-12 19:04:38,799 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188708799 2023-07-12 19:04:38,800 DEBUG [Listener at localhost/34615] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:38,800 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_CREATE_FS_LAYOUT; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:38,803 DEBUG [Listener at localhost/34615] assignment.RegionStateStore(147): Load hbase:meta entry region=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:38,803 INFO [Listener at localhost/34615] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:38,803 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188738803 2023-07-12 19:04:38,803 INFO [Listener at localhost/34615] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188798803 2023-07-12 19:04:38,803 INFO [Listener at localhost/34615] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:38,808 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:38,808 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:38,808 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:38,808 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:38,809 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e empty. 2023-07-12 19:04:38,809 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 empty. 2023-07-12 19:04:38,809 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 empty. 2023-07-12 19:04:38,810 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 empty. 2023-07-12 19:04:38,810 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:38,810 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:38,810 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:38,811 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:38,811 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testTruncateWithFailoverAtStep4 regions 2023-07-12 19:04:39,218 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp/data/default/testTruncateWithFailoverAtStep4/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:39,220 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(7675): creating {ENCODED => 602be7fec076ee1be772900bc4be2980, NAME => 'testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:39,220 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(7675): creating {ENCODED => f78c8d9fe947191cb6c8b3d8acef835e, NAME => 'testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:39,220 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 5d792f1fb61887b34b7931e503a40374, NAME => 'testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:39,629 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:39,629 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:39,629 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 5d792f1fb61887b34b7931e503a40374, disabling compactions & flushes 2023-07-12 19:04:39,629 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1604): Closing 602be7fec076ee1be772900bc4be2980, disabling compactions & flushes 2023-07-12 19:04:39,630 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1604): Closing f78c8d9fe947191cb6c8b3d8acef835e, disabling compactions & flushes 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. after waiting 0 ms 2023-07-12 19:04:39,630 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. after waiting 0 ms 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:39,630 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:39,630 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:39,630 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 5d792f1fb61887b34b7931e503a40374: 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:39,630 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-2] regionserver.HRegion(1558): Region close journal for 602be7fec076ee1be772900bc4be2980: 2023-07-12 19:04:39,631 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. after waiting 0 ms 2023-07-12 19:04:39,631 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:39,631 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(7675): creating {ENCODED => 5c1f0e8b4afac0d7d73b1239993f8752, NAME => 'testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testTruncateWithFailoverAtStep4', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.tmp 2023-07-12 19:04:39,631 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:39,631 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-0] regionserver.HRegion(1558): Region close journal for f78c8d9fe947191cb6c8b3d8acef835e: 2023-07-12 19:04:40,037 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:40,037 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1604): Closing 5c1f0e8b4afac0d7d73b1239993f8752, disabling compactions & flushes 2023-07-12 19:04:40,038 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:40,038 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:40,038 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. after waiting 0 ms 2023-07-12 19:04:40,038 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:40,038 INFO [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:40,038 DEBUG [RegionOpenAndInit-testTruncateWithFailoverAtStep4-pool-1] regionserver.HRegion(1558): Region close journal for 5c1f0e8b4afac0d7d73b1239993f8752: 2023-07-12 19:04:40,039 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:40,042 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188680042"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188680042"}]},"ts":"1689188680042"} 2023-07-12 19:04:40,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188680042"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188680042"}]},"ts":"1689188680042"} 2023-07-12 19:04:40,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188680042"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188680042"}]},"ts":"1689188680042"} 2023-07-12 19:04:40,043 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188680042"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188680042"}]},"ts":"1689188680042"} 2023-07-12 19:04:40,045 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 19:04:40,045 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:40,046 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:40,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:40,046 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:TRUNCATE_TABLE_ASSIGN_REGIONS, locked=true; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:40,053 INFO [Listener at localhost/34615] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 19:04:40,054 INFO [Listener at localhost/34615] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:40,054 ERROR [Listener at localhost/34615] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 19:04:40,054 ERROR [Listener at localhost/34615] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase4.apache.org,38203,1689188667859: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailoverAtStep(TestMasterFailoverWithProcedures.java:201) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testTruncateWithFailover(TestMasterFailoverWithProcedures.java:175) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:40,054 INFO [Listener at localhost/34615] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,38203,1689188667859' ***** 2023-07-12 19:04:40,054 INFO [Listener at localhost/34615] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/34615 2023-07-12 19:04:40,055 DEBUG [M:0;jenkins-hbase4:38203] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53ebc962, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:40,055 INFO [M:0;jenkins-hbase4:38203] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,38203,1689188667859 2023-07-12 19:04:40,055 DEBUG [M:0;jenkins-hbase4:38203] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6abbb7eb to 127.0.0.1:61051 2023-07-12 19:04:40,055 DEBUG [M:0;jenkins-hbase4:38203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:40,056 INFO [M:0;jenkins-hbase4:38203] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,38203,1689188667859; all regions closed. 2023-07-12 19:04:40,056 DEBUG [M:0;jenkins-hbase4:38203] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:40,056 DEBUG [M:0;jenkins-hbase4:38203] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:40,056 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:40,056 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188668855] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188668855,5,FailOnTimeoutGroup] 2023-07-12 19:04:40,056 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188668855] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188668855,5,FailOnTimeoutGroup] 2023-07-12 19:04:40,056 DEBUG [M:0;jenkins-hbase4:38203] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:40,056 INFO [M:0;jenkins-hbase4:38203] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:40,056 INFO [M:0;jenkins-hbase4:38203] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:40,057 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,057 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,057 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,057 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:40,057 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:40,057 INFO [M:0;jenkins-hbase4:38203] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:40,057 DEBUG [M:0;jenkins-hbase4:38203] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:40,057 INFO [M:0;jenkins-hbase4:38203] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:40,058 ERROR [M:0;jenkins-hbase4:38203] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:40,058 INFO [M:0;jenkins-hbase4:38203] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 19:04:40,058 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:40,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:40,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:40,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:40,059 DEBUG [M:0;jenkins-hbase4:38203] zookeeper.ZKUtil(398): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:40,059 WARN [M:0;jenkins-hbase4:38203] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:40,059 INFO [M:0;jenkins-hbase4:38203] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:40,060 INFO [M:0;jenkins-hbase4:38203] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 19:04:40,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:40,060 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 19:04:40,062 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,062 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,062 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:40,062 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:40,062 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:40,063 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:40,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,44039,1689188667887 from backup master directory 2023-07-12 19:04:40,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:40,064 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 not finished, retry = 0 2023-07-12 19:04:40,064 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:40,064 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:40,064 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:40,064 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:40,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:40,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:40,165 DEBUG [M:0;jenkins-hbase4:38203] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:40,165 INFO [M:0;jenkins-hbase4:38203] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:40,165 DEBUG [M:0;jenkins-hbase4:38203] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:40,165 DEBUG [M:0;jenkins-hbase4:38203] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:40,165 DEBUG [M:0;jenkins-hbase4:38203] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:40,165 ERROR [M:0;jenkins-hbase4:38203] regionserver.HRegion(1825): Memstore data size is 92212 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:40,165 INFO [M:0;jenkins-hbase4:38203] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:40,165 DEBUG [M:0;jenkins-hbase4:38203] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:40,165 INFO [M:0;jenkins-hbase4:38203] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:40,165 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:40,166 INFO [M:0;jenkins-hbase4:38203] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38203 2023-07-12 19:04:40,170 DEBUG [M:0;jenkins-hbase4:38203] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,38203,1689188667859 already deleted, retry=false 2023-07-12 19:04:40,263 WARN [RS-EventLoopGroup-14-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:38203 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38203 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:40,263 DEBUG [RS-EventLoopGroup-14-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:38203 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:38203 2023-07-12 19:04:40,271 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:40,272 INFO [M:0;jenkins-hbase4:38203] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,38203,1689188667859; zookeeper connection closed. 2023-07-12 19:04:40,272 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:38203-0x1015aec13380000, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:40,364 DEBUG [RS:0;jenkins-hbase4:39351] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:40,366 INFO [RS-EventLoopGroup-13-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56873, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:40,366 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,468 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x3ff53a97 to 127.0.0.1:61051 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:40,497 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1470266a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:40,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:40,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:40,498 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:40,503 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859-dead as it is dead 2023-07-12 19:04:40,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859-dead/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:40,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859-dead/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 after 0ms 2023-07-12 19:04:40,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859-dead/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 to hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:40,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,38203,1689188667859-dead 2023-07-12 19:04:40,505 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:40,507 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44039%2C1689188667887, suffix=, logDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,44039,1689188667887, archiveDir=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:40,521 DEBUG [RS-EventLoopGroup-14-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK] 2023-07-12 19:04:40,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/WALs/jenkins-hbase4.apache.org,44039,1689188667887/jenkins-hbase4.apache.org%2C44039%2C1689188667887.1689188680507 2023-07-12 19:04:40,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35881,DS-b79f8327-1c99-4b46-ba1f-ee5250908e62,DISK]] 2023-07-12 19:04:40,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:40,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:40,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:40,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:40,525 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:40,525 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:40,526 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:40,526 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:40,527 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 19:04:40,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:40,535 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 232, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=194, path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:40,536 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=90.05 KB heapSize=107.48 KB 2023-07-12 19:04:40,570 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,672 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,774 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,876 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=90.05 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c007fa4409f44889be20ee1b87c9a60a 2023-07-12 19:04:40,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c007fa4409f44889be20ee1b87c9a60a as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c007fa4409f44889be20ee1b87c9a60a 2023-07-12 19:04:40,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c007fa4409f44889be20ee1b87c9a60a, entries=232, sequenceid=194, filesize=95.3 K 2023-07-12 19:04:40,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~90.05 KB/92212, heapSize ~107.47 KB/110048, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 420ms, sequenceid=194, compaction requested=false; wal=null 2023-07-12 19:04:40,956 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C38203%2C1689188667859.1689188668783 2023-07-12 19:04:40,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:40,960 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/194.seqid, newMaxSeqId=194, maxSeqId=1 2023-07-12 19:04:40,961 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=195; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66511883, jitterRate=-0.008895710110664368}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:40,961 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:40,962 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:40,963 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:40,963 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:40,963 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:40,964 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:40,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.TruncateTableProcedure) stack ids=[0, 1, 2, 3] 2023-07-12 19:04:40,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:40,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:40,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:40,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:40,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:40,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testTruncateWithFailoverAtStep4 2023-07-12 19:04:40,970 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:40,971 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:40,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-07-12 19:04:40,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:40,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:40,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,39351,1689188667910, table=hbase:meta, region=1588230740 2023-07-12 19:04:40,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:40,974 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:40,978 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=44039] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:40,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:40,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:40,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:40,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:40,992 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:40,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:40,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:40,994 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:40,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,44039,1689188667887, sessionid=0x1015aec13380001, setting cluster-up flag (Was=true) 2023-07-12 19:04:40,997 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:40,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:41,003 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:41,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:41,006 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/.hbase-snapshot/.tmp 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:41,012 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:41,017 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188711017 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:41,018 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:41,019 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,019 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:TRUNCATE_TABLE_ADD_TO_META; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) 2023-07-12 19:04:41,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:41,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:41,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:41,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:41,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:41,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188681028,5,FailOnTimeoutGroup] 2023-07-12 19:04:41,028 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188681028,5,FailOnTimeoutGroup] 2023-07-12 19:04:41,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188681029, completionTime=-1 2023-07-12 19:04:41,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:41,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:41,034 DEBUG [hconnection-0x425dddb0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:41,035 INFO [RS-EventLoopGroup-14-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46128, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:41,037 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=4e5be26357fae50a1ff6b45ead0d6fbb, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,39351,1689188667910, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910, openSeqNum=2 2023-07-12 19:04:41,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=f78c8d9fe947191cb6c8b3d8acef835e, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:41,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=5d792f1fb61887b34b7931e503a40374, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:41,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=602be7fec076ee1be772900bc4be2980, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:41,038 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=5c1f0e8b4afac0d7d73b1239993f8752, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=-1 2023-07-12 19:04:41,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:41,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188741039 2023-07-12 19:04:41,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188801039 2023-07-12 19:04:41,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 9 msec 2023-07-12 19:04:41,045 WARN [master/jenkins-hbase4:0:becomeActiveMaster] master.TableStateManager(233): testTruncateWithFailoverAtStep4 has no table state in hbase:meta, assuming ENABLED 2023-07-12 19:04:41,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188681045"}]},"ts":"1689188681045"} 2023-07-12 19:04:41,047 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 19:04:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44039,1689188667887-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44039,1689188667887-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44039,1689188667887-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:44039, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:41,054 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 19:04:41,058 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:41,067 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:41,078 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 1.014sec 2023-07-12 19:04:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:41,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44039,1689188667887-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:41,079 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44039,1689188667887-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:41,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188681083"}]},"ts":"1689188681083"} 2023-07-12 19:04:41,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188681083"}]},"ts":"1689188681083"} 2023-07-12 19:04:41,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188681083"}]},"ts":"1689188681083"} 2023-07-12 19:04:41,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188681083"}]},"ts":"1689188681083"} 2023-07-12 19:04:41,090 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:41,090 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 19:04:41,093 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188681093"}]},"ts":"1689188681093"} 2023-07-12 19:04:41,095 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLING in hbase:meta 2023-07-12 19:04:41,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=f78c8d9fe947191cb6c8b3d8acef835e, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5d792f1fb61887b34b7931e503a40374, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=602be7fec076ee1be772900bc4be2980, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5c1f0e8b4afac0d7d73b1239993f8752, ASSIGN}] 2023-07-12 19:04:41,102 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5c1f0e8b4afac0d7d73b1239993f8752, ASSIGN 2023-07-12 19:04:41,102 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5c1f0e8b4afac0d7d73b1239993f8752, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:41,103 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=602be7fec076ee1be772900bc4be2980, ASSIGN 2023-07-12 19:04:41,104 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=602be7fec076ee1be772900bc4be2980, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:41,104 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5d792f1fb61887b34b7931e503a40374, ASSIGN 2023-07-12 19:04:41,105 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5d792f1fb61887b34b7931e503a40374, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:41,106 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=f78c8d9fe947191cb6c8b3d8acef835e, ASSIGN 2023-07-12 19:04:41,106 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=f78c8d9fe947191cb6c8b3d8acef835e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39351,1689188667910; forceNewPlan=false, retain=false 2023-07-12 19:04:41,255 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f78c8d9fe947191cb6c8b3d8acef835e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,255 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188681254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188681254"}]},"ts":"1689188681254"} 2023-07-12 19:04:41,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=28, state=RUNNABLE; OpenRegionProcedure f78c8d9fe947191cb6c8b3d8acef835e, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:41,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=602be7fec076ee1be772900bc4be2980, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188681260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188681260"}]},"ts":"1689188681260"} 2023-07-12 19:04:41,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=30, state=RUNNABLE; OpenRegionProcedure 602be7fec076ee1be772900bc4be2980, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:41,266 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=5c1f0e8b4afac0d7d73b1239993f8752, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,266 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681266"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188681266"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188681266"}]},"ts":"1689188681266"} 2023-07-12 19:04:41,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=31, state=RUNNABLE; OpenRegionProcedure 5c1f0e8b4afac0d7d73b1239993f8752, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:41,272 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5d792f1fb61887b34b7931e503a40374, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,272 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681272"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188681272"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188681272"}]},"ts":"1689188681272"} 2023-07-12 19:04:41,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=29, state=RUNNABLE; OpenRegionProcedure 5d792f1fb61887b34b7931e503a40374, server=jenkins-hbase4.apache.org,39351,1689188667910}] 2023-07-12 19:04:41,410 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,410 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:41,412 INFO [RS-EventLoopGroup-14-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:41,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f78c8d9fe947191cb6c8b3d8acef835e, NAME => 'testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 19:04:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,417 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,418 DEBUG [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f1 2023-07-12 19:04:41,418 DEBUG [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f1 2023-07-12 19:04:41,418 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f78c8d9fe947191cb6c8b3d8acef835e columnFamilyName f1 2023-07-12 19:04:41,419 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] regionserver.HStore(310): Store=f78c8d9fe947191cb6c8b3d8acef835e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,419 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,420 DEBUG [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f2 2023-07-12 19:04:41,420 DEBUG [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f2 2023-07-12 19:04:41,420 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f78c8d9fe947191cb6c8b3d8acef835e columnFamilyName f2 2023-07-12 19:04:41,421 INFO [StoreOpener-f78c8d9fe947191cb6c8b3d8acef835e-1] regionserver.HStore(310): Store=f78c8d9fe947191cb6c8b3d8acef835e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:41,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:41,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:41,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f78c8d9fe947191cb6c8b3d8acef835e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63541553, jitterRate=-0.053157076239585876}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:41,426 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f78c8d9fe947191cb6c8b3d8acef835e: 2023-07-12 19:04:41,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e., pid=32, masterSystemTime=1689188681410 2023-07-12 19:04:41,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:41,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:41,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:41,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 602be7fec076ee1be772900bc4be2980, NAME => 'testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 19:04:41,429 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=f78c8d9fe947191cb6c8b3d8acef835e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,430 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681429"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188681429"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188681429"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188681429"}]},"ts":"1689188681429"} 2023-07-12 19:04:41,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:41,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,431 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,432 DEBUG [StoreOpener-602be7fec076ee1be772900bc4be2980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f1 2023-07-12 19:04:41,432 DEBUG [StoreOpener-602be7fec076ee1be772900bc4be2980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f1 2023-07-12 19:04:41,432 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 602be7fec076ee1be772900bc4be2980 columnFamilyName f1 2023-07-12 19:04:41,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-07-12 19:04:41,433 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; OpenRegionProcedure f78c8d9fe947191cb6c8b3d8acef835e, server=jenkins-hbase4.apache.org,39351,1689188667910 in 174 msec 2023-07-12 19:04:41,433 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] regionserver.HStore(310): Store=602be7fec076ee1be772900bc4be2980/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,433 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=f78c8d9fe947191cb6c8b3d8acef835e, ASSIGN in 333 msec 2023-07-12 19:04:41,434 DEBUG [StoreOpener-602be7fec076ee1be772900bc4be2980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f2 2023-07-12 19:04:41,434 DEBUG [StoreOpener-602be7fec076ee1be772900bc4be2980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f2 2023-07-12 19:04:41,434 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 602be7fec076ee1be772900bc4be2980 columnFamilyName f2 2023-07-12 19:04:41,435 INFO [StoreOpener-602be7fec076ee1be772900bc4be2980-1] regionserver.HStore(310): Store=602be7fec076ee1be772900bc4be2980/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:41,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:41,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:41,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 602be7fec076ee1be772900bc4be2980; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59830992, jitterRate=-0.10844874382019043}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:41,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 602be7fec076ee1be772900bc4be2980: 2023-07-12 19:04:41,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980., pid=33, masterSystemTime=1689188681410 2023-07-12 19:04:41,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:41,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:41,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:41,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5d792f1fb61887b34b7931e503a40374, NAME => 'testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 19:04:41,442 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=602be7fec076ee1be772900bc4be2980, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,442 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,442 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681442"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188681442"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188681442"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188681442"}]},"ts":"1689188681442"} 2023-07-12 19:04:41,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:41,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,444 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,445 DEBUG [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f1 2023-07-12 19:04:41,445 DEBUG [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f1 2023-07-12 19:04:41,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=30 2023-07-12 19:04:41,445 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5d792f1fb61887b34b7931e503a40374 columnFamilyName f1 2023-07-12 19:04:41,445 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=30, state=SUCCESS; OpenRegionProcedure 602be7fec076ee1be772900bc4be2980, server=jenkins-hbase4.apache.org,39351,1689188667910 in 182 msec 2023-07-12 19:04:41,446 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] regionserver.HStore(310): Store=5d792f1fb61887b34b7931e503a40374/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,446 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,447 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=602be7fec076ee1be772900bc4be2980, ASSIGN in 346 msec 2023-07-12 19:04:41,447 DEBUG [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f2 2023-07-12 19:04:41,447 DEBUG [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f2 2023-07-12 19:04:41,447 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5d792f1fb61887b34b7931e503a40374 columnFamilyName f2 2023-07-12 19:04:41,448 INFO [StoreOpener-5d792f1fb61887b34b7931e503a40374-1] regionserver.HStore(310): Store=5d792f1fb61887b34b7931e503a40374/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,448 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,449 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:41,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:41,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:41,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5d792f1fb61887b34b7931e503a40374; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69038085, jitterRate=0.02874763309955597}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:41,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5d792f1fb61887b34b7931e503a40374: 2023-07-12 19:04:41,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374., pid=35, masterSystemTime=1689188681410 2023-07-12 19:04:41,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:41,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:41,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:41,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5c1f0e8b4afac0d7d73b1239993f8752, NAME => 'testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 19:04:41,456 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=5d792f1fb61887b34b7931e503a40374, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testTruncateWithFailoverAtStep4 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,456 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1689188681455"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188681455"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188681455"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188681455"}]},"ts":"1689188681455"} 2023-07-12 19:04:41,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:41,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,456 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,457 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,458 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=29 2023-07-12 19:04:41,459 DEBUG [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f1 2023-07-12 19:04:41,459 DEBUG [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f1 2023-07-12 19:04:41,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=29, state=SUCCESS; OpenRegionProcedure 5d792f1fb61887b34b7931e503a40374, server=jenkins-hbase4.apache.org,39351,1689188667910 in 184 msec 2023-07-12 19:04:41,459 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c1f0e8b4afac0d7d73b1239993f8752 columnFamilyName f1 2023-07-12 19:04:41,459 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] regionserver.HStore(310): Store=5c1f0e8b4afac0d7d73b1239993f8752/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5d792f1fb61887b34b7931e503a40374, ASSIGN in 359 msec 2023-07-12 19:04:41,460 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,461 DEBUG [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f2 2023-07-12 19:04:41,461 DEBUG [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f2 2023-07-12 19:04:41,461 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5c1f0e8b4afac0d7d73b1239993f8752 columnFamilyName f2 2023-07-12 19:04:41,462 INFO [StoreOpener-5c1f0e8b4afac0d7d73b1239993f8752-1] regionserver.HStore(310): Store=5c1f0e8b4afac0d7d73b1239993f8752/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:41,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testTruncateWithFailoverAtStep4 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:41,465 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:41,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:41,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5c1f0e8b4afac0d7d73b1239993f8752; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70650918, jitterRate=0.052780717611312866}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:41,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5c1f0e8b4afac0d7d73b1239993f8752: 2023-07-12 19:04:41,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752., pid=34, masterSystemTime=1689188681410 2023-07-12 19:04:41,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:41,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:41,469 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=5c1f0e8b4afac0d7d73b1239993f8752, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,470 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1689188681469"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188681469"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188681469"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188681469"}]},"ts":"1689188681469"} 2023-07-12 19:04:41,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=31 2023-07-12 19:04:41,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=31, state=SUCCESS; OpenRegionProcedure 5c1f0e8b4afac0d7d73b1239993f8752, server=jenkins-hbase4.apache.org,39351,1689188667910 in 202 msec 2023-07-12 19:04:41,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=27 2023-07-12 19:04:41,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testTruncateWithFailoverAtStep4, region=5c1f0e8b4afac0d7d73b1239993f8752, ASSIGN in 372 msec 2023-07-12 19:04:41,474 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testTruncateWithFailoverAtStep4","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188681474"}]},"ts":"1689188681474"} 2023-07-12 19:04:41,475 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testTruncateWithFailoverAtStep4, state=ENABLED in hbase:meta 2023-07-12 19:04:41,477 DEBUG [PEWorker-1] procedure.TruncateTableProcedure(145): truncate 'testTruncateWithFailoverAtStep4' completed 2023-07-12 19:04:41,478 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; TruncateTableProcedure (table=testTruncateWithFailoverAtStep4 preserveSplits=true) in 4.7240 sec 2023-07-12 19:04:41,557 WARN [Listener at localhost/34615] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:41,558 DEBUG [Listener at localhost/34615] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testTruncateWithFailoverAtStep4 get assigned. Timeout = 60000ms 2023-07-12 19:04:41,559 INFO [Listener at localhost/34615] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:04:41,563 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(3484): All regions for table testTruncateWithFailoverAtStep4 assigned to meta. Checking AM states. 2023-07-12 19:04:41,563 INFO [Listener at localhost/34615] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-12 19:04:41,563 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(3504): All regions for table testTruncateWithFailoverAtStep4 assigned. 2023-07-12 19:04:41,567 DEBUG [Listener at localhost/34615] util.CommonFSUtils(682): File system contents for path hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4 2023-07-12 19:04:41,567 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-.tabledesc/ 2023-07-12 19:04:41,568 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |----.tableinfo.0000000001 2023-07-12 19:04:41,568 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-.tmp/ 2023-07-12 19:04:41,568 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-5c1f0e8b4afac0d7d73b1239993f8752/ 2023-07-12 19:04:41,569 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:41,569 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:41,569 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:41,569 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:41,570 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:41,570 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-5d792f1fb61887b34b7931e503a40374/ 2023-07-12 19:04:41,570 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:41,570 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:41,571 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:41,571 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:41,571 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:41,572 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-602be7fec076ee1be772900bc4be2980/ 2023-07-12 19:04:41,572 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:41,572 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:41,572 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:41,573 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:41,573 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:41,573 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |-f78c8d9fe947191cb6c8b3d8acef835e/ 2023-07-12 19:04:41,573 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |----.regioninfo 2023-07-12 19:04:41,574 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f1/ 2023-07-12 19:04:41,574 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----f2/ 2023-07-12 19:04:41,574 DEBUG [Listener at localhost/34615] util.CommonFSUtils(699): |----recovered.edits/ 2023-07-12 19:04:41,575 DEBUG [Listener at localhost/34615] util.CommonFSUtils(702): |-------1.seqid 2023-07-12 19:04:41,582 DEBUG [Listener at localhost/34615] procedure.MasterProcedureTestingUtility(212): Table directory layout is as expected. 2023-07-12 19:04:41,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Scan size: 154 connection: 172.31.14.131:33166 deadline: 1689188741586, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,,1689188671339.5bb0a9498878cf2bd91ed66e846a7170. is not online on jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,696 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 19:04:41,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Scan size: 158 connection: 172.31.14.131:33166 deadline: 1689188741696, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,a,1689188671339.fc6cf34c3ef9229ba8327618b2701f07. is not online on jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,801 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 19:04:41,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Scan size: 158 connection: 172.31.14.131:33166 deadline: 1689188741802, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,b,1689188671339.79113b41be6b3e023e187f2b1fd17460. is not online on jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:41,908 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 19:04:41,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Scan size: 158 connection: 172.31.14.131:33166 deadline: 1689188741908, exception=org.apache.hadoop.hbase.NotServingRegionException: testTruncateWithFailoverAtStep4,c,1689188671339.1a12a39783f6f1f6efd1a03e6dd6a9a3. is not online on jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:42,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:42,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:42,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:42,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39351] regionserver.HRegion(8805): writing data to region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. with WAL disabled. Data may be lost in the event of a crash. 2023-07-12 19:04:42,023 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'a', inclusive 2023-07-12 19:04:42,024 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'b', inclusive 2023-07-12 19:04:42,025 DEBUG [Listener at localhost/34615] client.ClientScanner(240): Advancing internal scanner to startKey at 'c', inclusive 2023-07-12 19:04:42,026 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:04:42,027 INFO [Listener at localhost/34615] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 19:04:42,027 DEBUG [Listener at localhost/34615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1088e925 to 127.0.0.1:61051 2023-07-12 19:04:42,027 DEBUG [Listener at localhost/34615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:42,027 DEBUG [Listener at localhost/34615] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:04:42,027 DEBUG [Listener at localhost/34615] util.JVMClusterUtil(257): Found active master hash=1906152699, stopped=false 2023-07-12 19:04:42,027 INFO [Listener at localhost/34615] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:42,029 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:42,029 INFO [Listener at localhost/34615] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:42,029 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:42,029 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:42,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:42,030 DEBUG [Listener at localhost/34615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ff53a97 to 127.0.0.1:61051 2023-07-12 19:04:42,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:42,030 DEBUG [Listener at localhost/34615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:42,030 INFO [Listener at localhost/34615] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39351,1689188667910' ***** 2023-07-12 19:04:42,030 INFO [Listener at localhost/34615] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:04:42,030 INFO [RS:0;jenkins-hbase4:39351] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:04:42,030 INFO [RS:0;jenkins-hbase4:39351] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:04:42,030 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:04:42,030 INFO [RS:0;jenkins-hbase4:39351] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for 4e5be26357fae50a1ff6b45ead0d6fbb 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:42,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4e5be26357fae50a1ff6b45ead0d6fbb, disabling compactions & flushes 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for 5c1f0e8b4afac0d7d73b1239993f8752 2023-07-12 19:04:42,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:42,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:42,031 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:42,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. after waiting 0 ms 2023-07-12 19:04:42,031 DEBUG [RS:0;jenkins-hbase4:39351] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x29e5211d to 127.0.0.1:61051 2023-07-12 19:04:42,032 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:42,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 4e5be26357fae50a1ff6b45ead0d6fbb 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 19:04:42,032 DEBUG [RS:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:42,032 INFO [RS:0;jenkins-hbase4:39351] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:04:42,032 INFO [RS:0;jenkins-hbase4:39351] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:04:42,032 INFO [RS:0;jenkins-hbase4:39351] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:04:42,032 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:04:42,032 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-12 19:04:42,032 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1478): Online Regions={4e5be26357fae50a1ff6b45ead0d6fbb=hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb., f78c8d9fe947191cb6c8b3d8acef835e=testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e., 602be7fec076ee1be772900bc4be2980=testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980., 1588230740=hbase:meta,,1.1588230740, 5c1f0e8b4afac0d7d73b1239993f8752=testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752., 5d792f1fb61887b34b7931e503a40374=testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.} 2023-07-12 19:04:42,032 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:42,032 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 4e5be26357fae50a1ff6b45ead0d6fbb, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,032 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:42,032 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:42,032 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:42,032 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:42,033 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=18.80 KB heapSize=30.18 KB 2023-07-12 19:04:42,075 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:42,232 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 4e5be26357fae50a1ff6b45ead0d6fbb, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,427 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bd1dd7768a82675: from storage DS-b79f8327-1c99-4b46-ba1f-ee5250908e62 node DatanodeRegistration(127.0.0.1:35881, datanodeUuid=9642a675-1733-4f54-9789-2d4949d4bae8, infoPort=43795, infoSecurePort=0, ipcPort=34615, storageInfo=lv=-57;cid=testClusterID;nsid=1941602023;c=1689188667032), blocks: 13, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 19:04:42,427 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bd1dd7768a82675: from storage DS-3cb6b2d0-7c23-46fd-8897-fe670ca21840 node DatanodeRegistration(127.0.0.1:35881, datanodeUuid=9642a675-1733-4f54-9789-2d4949d4bae8, infoPort=43795, infoSecurePort=0, ipcPort=34615, storageInfo=lv=-57;cid=testClusterID;nsid=1941602023;c=1689188667032), blocks: 14, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:42,433 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 4e5be26357fae50a1ff6b45ead0d6fbb, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/.tmp/info/3bc6fe8bc7614642a1a303659940caf4 2023-07-12 19:04:42,440 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.31 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/info/0dcb71c100144cb2a47e8c252a6abdac 2023-07-12 19:04:42,447 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dcb71c100144cb2a47e8c252a6abdac 2023-07-12 19:04:42,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/.tmp/info/3bc6fe8bc7614642a1a303659940caf4 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/info/3bc6fe8bc7614642a1a303659940caf4 2023-07-12 19:04:42,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/info/3bc6fe8bc7614642a1a303659940caf4, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 19:04:42,454 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 4e5be26357fae50a1ff6b45ead0d6fbb in 422ms, sequenceid=6, compaction requested=false 2023-07-12 19:04:42,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/namespace/4e5be26357fae50a1ff6b45ead0d6fbb/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 19:04:42,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4e5be26357fae50a1ff6b45ead0d6fbb: 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689188670065.4e5be26357fae50a1ff6b45ead0d6fbb. 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f78c8d9fe947191cb6c8b3d8acef835e, disabling compactions & flushes 2023-07-12 19:04:42,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. after waiting 0 ms 2023-07-12 19:04:42,463 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:42,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f78c8d9fe947191cb6c8b3d8acef835e 2/2 column families, dataSize=3.66 KB heapSize=8.38 KB 2023-07-12 19:04:42,633 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,833 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:42,861 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=447 B at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/rep_barrier/5854d000a3154191a8be7b04a5fd0b21 2023-07-12 19:04:42,866 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5854d000a3154191a8be7b04a5fd0b21 2023-07-12 19:04:42,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.83 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/.tmp/f1/5562060431944c89b7fab920a210e3e7 2023-07-12 19:04:42,965 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 19:04:42,965 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 19:04:43,034 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1474): Waiting on 5 regions to close 2023-07-12 19:04:43,034 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1478): Online Regions={f78c8d9fe947191cb6c8b3d8acef835e=testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e., 602be7fec076ee1be772900bc4be2980=testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980., 1588230740=hbase:meta,,1.1588230740, 5c1f0e8b4afac0d7d73b1239993f8752=testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752., 5d792f1fb61887b34b7931e503a40374=testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.} 2023-07-12 19:04:43,034 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:43,234 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 1588230740, 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980, f78c8d9fe947191cb6c8b3d8acef835e 2023-07-12 19:04:43,274 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.05 KB at sequenceid=47 (bloomFilter=false), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/table/0dabd5ecbf1e4ab3bee426320fb24cf8 2023-07-12 19:04:43,280 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dabd5ecbf1e4ab3bee426320fb24cf8 2023-07-12 19:04:43,280 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/info/0dcb71c100144cb2a47e8c252a6abdac as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/info/0dcb71c100144cb2a47e8c252a6abdac 2023-07-12 19:04:43,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.83 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/.tmp/f2/0614f0a57d204c4ea5480e23d057b5a1 2023-07-12 19:04:43,285 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dcb71c100144cb2a47e8c252a6abdac 2023-07-12 19:04:43,285 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/info/0dcb71c100144cb2a47e8c252a6abdac, entries=58, sequenceid=47, filesize=12.6 K 2023-07-12 19:04:43,286 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/rep_barrier/5854d000a3154191a8be7b04a5fd0b21 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/rep_barrier/5854d000a3154191a8be7b04a5fd0b21 2023-07-12 19:04:43,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/.tmp/f1/5562060431944c89b7fab920a210e3e7 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f1/5562060431944c89b7fab920a210e3e7 2023-07-12 19:04:43,292 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5854d000a3154191a8be7b04a5fd0b21 2023-07-12 19:04:43,292 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/rep_barrier/5854d000a3154191a8be7b04a5fd0b21, entries=4, sequenceid=47, filesize=5.4 K 2023-07-12 19:04:43,293 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/.tmp/table/0dabd5ecbf1e4ab3bee426320fb24cf8 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/table/0dabd5ecbf1e4ab3bee426320fb24cf8 2023-07-12 19:04:43,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f1/5562060431944c89b7fab920a210e3e7, entries=28, sequenceid=5, filesize=6.7 K 2023-07-12 19:04:43,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/.tmp/f2/0614f0a57d204c4ea5480e23d057b5a1 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f2/0614f0a57d204c4ea5480e23d057b5a1 2023-07-12 19:04:43,300 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0dabd5ecbf1e4ab3bee426320fb24cf8 2023-07-12 19:04:43,300 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/table/0dabd5ecbf1e4ab3bee426320fb24cf8, entries=11, sequenceid=47, filesize=5.7 K 2023-07-12 19:04:43,301 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~18.80 KB/19256, heapSize ~30.13 KB/30856, currentSize=0 B/0 for 1588230740 in 1268ms, sequenceid=47, compaction requested=false 2023-07-12 19:04:43,304 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/f2/0614f0a57d204c4ea5480e23d057b5a1, entries=28, sequenceid=5, filesize=6.7 K 2023-07-12 19:04:43,305 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.66 KB/3752, heapSize ~8.34 KB/8544, currentSize=0 B/0 for f78c8d9fe947191cb6c8b3d8acef835e in 842ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:43,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testTruncateWithFailoverAtStep4' 2023-07-12 19:04:43,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/hbase/meta/1588230740/recovered.edits/50.seqid, newMaxSeqId=50, maxSeqId=1 2023-07-12 19:04:43,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/f78c8d9fe947191cb6c8b3d8acef835e/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:43,314 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:43,315 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:43,315 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:43,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f78c8d9fe947191cb6c8b3d8acef835e: 2023-07-12 19:04:43,315 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:43,315 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,,1689188677290.f78c8d9fe947191cb6c8b3d8acef835e. 2023-07-12 19:04:43,315 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:43,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 602be7fec076ee1be772900bc4be2980, disabling compactions & flushes 2023-07-12 19:04:43,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:43,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:43,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. after waiting 0 ms 2023-07-12 19:04:43,316 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:43,316 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 602be7fec076ee1be772900bc4be2980 2/2 column families, dataSize=532 B heapSize=1.63 KB 2023-07-12 19:04:43,434 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:43,635 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:43,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/.tmp/f1/cc2332395edc421d8d769d9174fcbf87 2023-07-12 19:04:43,835 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:44,035 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-12 19:04:44,035 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1478): Online Regions={602be7fec076ee1be772900bc4be2980=testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980., 5c1f0e8b4afac0d7d73b1239993f8752=testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752., 5d792f1fb61887b34b7931e503a40374=testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.} 2023-07-12 19:04:44,035 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374, 602be7fec076ee1be772900bc4be2980 2023-07-12 19:04:44,137 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=266 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/.tmp/f2/3d290446f2a24b29a2e194710f032785 2023-07-12 19:04:44,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/.tmp/f1/cc2332395edc421d8d769d9174fcbf87 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f1/cc2332395edc421d8d769d9174fcbf87 2023-07-12 19:04:44,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f1/cc2332395edc421d8d769d9174fcbf87, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 19:04:44,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/.tmp/f2/3d290446f2a24b29a2e194710f032785 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f2/3d290446f2a24b29a2e194710f032785 2023-07-12 19:04:44,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/f2/3d290446f2a24b29a2e194710f032785, entries=4, sequenceid=5, filesize=5.1 K 2023-07-12 19:04:44,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~532 B/532, heapSize ~1.59 KB/1632, currentSize=0 B/0 for 602be7fec076ee1be772900bc4be2980 in 838ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:44,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/602be7fec076ee1be772900bc4be2980/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:44,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 602be7fec076ee1be772900bc4be2980: 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,b,1689188677290.602be7fec076ee1be772900bc4be2980. 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5c1f0e8b4afac0d7d73b1239993f8752, disabling compactions & flushes 2023-07-12 19:04:44,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. after waiting 0 ms 2023-07-12 19:04:44,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:44,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5c1f0e8b4afac0d7d73b1239993f8752 2/2 column families, dataSize=2.09 KB heapSize=5 KB 2023-07-12 19:04:44,235 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:44,436 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:44,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.04 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/.tmp/f1/eecd0df3b6314af78c4b5c40eff1d58a 2023-07-12 19:04:44,636 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:44,836 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5c1f0e8b4afac0d7d73b1239993f8752, 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:44,984 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.04 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/.tmp/f2/3a105aa8a24e4cccb47897045a8a3c77 2023-07-12 19:04:44,990 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/.tmp/f1/eecd0df3b6314af78c4b5c40eff1d58a as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f1/eecd0df3b6314af78c4b5c40eff1d58a 2023-07-12 19:04:44,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f1/eecd0df3b6314af78c4b5c40eff1d58a, entries=16, sequenceid=5, filesize=5.9 K 2023-07-12 19:04:44,995 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/.tmp/f2/3a105aa8a24e4cccb47897045a8a3c77 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f2/3a105aa8a24e4cccb47897045a8a3c77 2023-07-12 19:04:45,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/f2/3a105aa8a24e4cccb47897045a8a3c77, entries=16, sequenceid=5, filesize=5.9 K 2023-07-12 19:04:45,000 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.09 KB/2140, heapSize ~4.97 KB/5088, currentSize=0 B/0 for 5c1f0e8b4afac0d7d73b1239993f8752 in 838ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5c1f0e8b4afac0d7d73b1239993f8752/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:45,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5c1f0e8b4afac0d7d73b1239993f8752: 2023-07-12 19:04:45,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,c,1689188677290.5c1f0e8b4afac0d7d73b1239993f8752. 2023-07-12 19:04:45,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5d792f1fb61887b34b7931e503a40374, disabling compactions & flushes 2023-07-12 19:04:45,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:45,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:45,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. after waiting 0 ms 2023-07-12 19:04:45,007 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:45,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5d792f1fb61887b34b7931e503a40374 2/2 column families, dataSize=264 B heapSize=1.06 KB 2023-07-12 19:04:45,036 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-12 19:04:45,037 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1478): Online Regions={5d792f1fb61887b34b7931e503a40374=testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374.} 2023-07-12 19:04:45,037 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:45,237 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:45,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=132 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/.tmp/f1/14ef875bec4e41dda578313ade87f00b 2023-07-12 19:04:45,437 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:45,637 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:45,829 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=132 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/.tmp/f2/94075306d28349a7b1dce094996ed39b 2023-07-12 19:04:45,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/.tmp/f1/14ef875bec4e41dda578313ade87f00b as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f1/14ef875bec4e41dda578313ade87f00b 2023-07-12 19:04:45,838 DEBUG [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1504): Waiting on 5d792f1fb61887b34b7931e503a40374 2023-07-12 19:04:45,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f1/14ef875bec4e41dda578313ade87f00b, entries=2, sequenceid=5, filesize=5.0 K 2023-07-12 19:04:45,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/.tmp/f2/94075306d28349a7b1dce094996ed39b as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f2/94075306d28349a7b1dce094996ed39b 2023-07-12 19:04:45,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/f2/94075306d28349a7b1dce094996ed39b, entries=2, sequenceid=5, filesize=5.0 K 2023-07-12 19:04:45,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~264 B/264, heapSize ~1.03 KB/1056, currentSize=0 B/0 for 5d792f1fb61887b34b7931e503a40374 in 839ms, sequenceid=5, compaction requested=false 2023-07-12 19:04:45,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/data/default/testTruncateWithFailoverAtStep4/5d792f1fb61887b34b7931e503a40374/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=1 2023-07-12 19:04:45,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:45,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5d792f1fb61887b34b7931e503a40374: 2023-07-12 19:04:45,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testTruncateWithFailoverAtStep4,a,1689188677290.5d792f1fb61887b34b7931e503a40374. 2023-07-12 19:04:45,987 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(152): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:46,038 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39351,1689188667910; all regions closed. 2023-07-12 19:04:46,041 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910/jenkins-hbase4.apache.org%2C39351%2C1689188667910.meta.1689188670010.meta not finished, retry = 0 2023-07-12 19:04:46,144 DEBUG [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/oldWALs 2023-07-12 19:04:46,144 INFO [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39351%2C1689188667910.meta:.meta(num 1689188670010) 2023-07-12 19:04:46,147 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/WALs/jenkins-hbase4.apache.org,39351,1689188667910/jenkins-hbase4.apache.org%2C39351%2C1689188667910.1689188669091 not finished, retry = 0 2023-07-12 19:04:46,249 DEBUG [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/oldWALs 2023-07-12 19:04:46,250 INFO [RS:0;jenkins-hbase4:39351] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39351%2C1689188667910:(num 1689188669091) 2023-07-12 19:04:46,250 DEBUG [RS:0;jenkins-hbase4:39351] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:46,250 INFO [RS:0;jenkins-hbase4:39351] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:46,250 INFO [RS:0;jenkins-hbase4:39351] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-12 19:04:46,250 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:46,251 INFO [RS:0;jenkins-hbase4:39351] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39351 2023-07-12 19:04:46,252 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:46,252 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39351,1689188667910 2023-07-12 19:04:46,253 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:46,254 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39351,1689188667910] 2023-07-12 19:04:46,254 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39351,1689188667910; numProcessing=1 2023-07-12 19:04:46,256 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39351,1689188667910 already deleted, retry=false 2023-07-12 19:04:46,256 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39351,1689188667910 expired; onlineServers=0 2023-07-12 19:04:46,256 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,44039,1689188667887' ***** 2023-07-12 19:04:46,256 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:04:46,256 DEBUG [M:1;jenkins-hbase4:44039] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75a21e10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:46,257 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,44039,1689188667887 2023-07-12 19:04:46,257 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,44039,1689188667887; all regions closed. 2023-07-12 19:04:46,257 DEBUG [M:1;jenkins-hbase4:44039] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:46,257 DEBUG [M:1;jenkins-hbase4:44039] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:46,257 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:46,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188681028] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188681028,5,FailOnTimeoutGroup] 2023-07-12 19:04:46,257 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188681028] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188681028,5,FailOnTimeoutGroup] 2023-07-12 19:04:46,257 DEBUG [M:1;jenkins-hbase4:44039] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:46,258 INFO [M:1;jenkins-hbase4:44039] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:46,258 INFO [M:1;jenkins-hbase4:44039] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:46,258 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:46,258 INFO [M:1;jenkins-hbase4:44039] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:46,258 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:46,258 DEBUG [M:1;jenkins-hbase4:44039] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:46,258 INFO [M:1;jenkins-hbase4:44039] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:46,259 ERROR [M:1;jenkins-hbase4:44039] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:46,259 INFO [M:1;jenkins-hbase4:44039] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:46,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:46,259 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:46,259 DEBUG [M:1;jenkins-hbase4:44039] zookeeper.ZKUtil(398): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:46,259 WARN [M:1;jenkins-hbase4:44039] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:46,259 INFO [M:1;jenkins-hbase4:44039] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:46,260 INFO [M:1;jenkins-hbase4:44039] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:04:46,260 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:46,260 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:46,260 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:46,260 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:46,260 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:46,260 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=26.01 KB heapSize=31.70 KB 2023-07-12 19:04:46,354 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:46,354 INFO [RS:0;jenkins-hbase4:39351] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39351,1689188667910; zookeeper connection closed. 2023-07-12 19:04:46,354 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): regionserver:39351-0x1015aec13380002, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:46,355 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@572f905] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@572f905 2023-07-12 19:04:46,355 INFO [Listener at localhost/34615] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 19:04:46,669 INFO [M:1;jenkins-hbase4:44039] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.01 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ab8e8e61475641c39b55b8c569c5d866 2023-07-12 19:04:46,675 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/ab8e8e61475641c39b55b8c569c5d866 as hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ab8e8e61475641c39b55b8c569c5d866 2023-07-12 19:04:46,681 INFO [M:1;jenkins-hbase4:44039] regionserver.HStore(1080): Added hdfs://localhost:40837/user/jenkins/test-data/c21a8803-5116-78ee-62ed-7fcf8dde6589/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/ab8e8e61475641c39b55b8c569c5d866, entries=9, sequenceid=253, filesize=5.4 K 2023-07-12 19:04:46,682 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegion(2948): Finished flush of dataSize ~26.01 KB/26634, heapSize ~31.68 KB/32440, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 422ms, sequenceid=253, compaction requested=false 2023-07-12 19:04:46,684 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:46,684 DEBUG [M:1;jenkins-hbase4:44039] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:46,687 INFO [M:1;jenkins-hbase4:44039] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:46,687 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:46,687 INFO [M:1;jenkins-hbase4:44039] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44039 2023-07-12 19:04:46,689 DEBUG [M:1;jenkins-hbase4:44039] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,44039,1689188667887 already deleted, retry=false 2023-07-12 19:04:46,791 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:46,791 INFO [M:1;jenkins-hbase4:44039] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,44039,1689188667887; zookeeper connection closed. 2023-07-12 19:04:46,791 DEBUG [Listener at localhost/34615-EventThread] zookeeper.ZKWatcher(600): master:44039-0x1015aec13380001, quorum=127.0.0.1:61051, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:46,792 WARN [Listener at localhost/34615] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:04:46,796 INFO [Listener at localhost/34615] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:46,900 WARN [BP-1509376613-172.31.14.131-1689188667032 heartbeating to localhost/127.0.0.1:40837] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:04:46,901 WARN [BP-1509376613-172.31.14.131-1689188667032 heartbeating to localhost/127.0.0.1:40837] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1509376613-172.31.14.131-1689188667032 (Datanode Uuid 9642a675-1733-4f54-9789-2d4949d4bae8) service to localhost/127.0.0.1:40837 2023-07-12 19:04:46,902 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c/dfs/data/data1/current/BP-1509376613-172.31.14.131-1689188667032] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:46,902 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/cluster_649b82e7-c8db-be5e-660c-955bf78cad3c/dfs/data/data2/current/BP-1509376613-172.31.14.131-1689188667032] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:46,913 INFO [Listener at localhost/34615] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:46,918 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:47,024 INFO [Listener at localhost/34615] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:04:47,035 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 19:04:47,047 INFO [Listener at localhost/34615] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testTruncateWithFailover Thread=124 (was 102) Potentially hanging thread: LeaseRenewer:jenkins@localhost:40837 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/34615 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-13-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:40837 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:40837 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:40837 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-14-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:40837 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-12-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=481 (was 443) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 273), ProcessCount=170 (was 170), AvailableMemoryMB=8245 (was 8349) 2023-07-12 19:04:47,056 INFO [Listener at localhost/34615] hbase.ResourceChecker(147): before: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=124, OpenFileDescriptor=481, MaxFileDescriptor=60000, SystemLoadAverage=209, ProcessCount=170, AvailableMemoryMB=8245 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=2, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/hadoop.log.dir so I do NOT create it in target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/55f3aa77-6ac4-0df8-94bc-e9b998c99288/hadoop.tmp.dir so I do NOT create it in target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3, deleteOnExit=true 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/test.cache.data in system properties and HBase conf 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/hadoop.tmp.dir in system properties and HBase conf 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/hadoop.log.dir in system properties and HBase conf 2023-07-12 19:04:47,057 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-12 19:04:47,058 DEBUG [Listener at localhost/34615] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-12 19:04:47,058 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/nfs.dump.dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/java.io.tmpdir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-12 19:04:47,059 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-12 19:04:47,060 INFO [Listener at localhost/34615] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-12 19:04:47,100 WARN [Listener at localhost/34615] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:47,100 WARN [Listener at localhost/34615] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:47,140 WARN [Listener at localhost/34615] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:47,142 INFO [Listener at localhost/34615] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:47,147 INFO [Listener at localhost/34615] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/java.io.tmpdir/Jetty_localhost_41583_hdfs____.4hh0qr/webapp 2023-07-12 19:04:47,241 INFO [Listener at localhost/34615] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41583 2023-07-12 19:04:47,245 WARN [Listener at localhost/34615] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-12 19:04:47,245 WARN [Listener at localhost/34615] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-12 19:04:47,290 WARN [Listener at localhost/33505] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:47,305 WARN [Listener at localhost/33505] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-12 19:04:47,307 WARN [Listener at localhost/33505] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-12 19:04:47,308 INFO [Listener at localhost/33505] log.Slf4jLog(67): jetty-6.1.26 2023-07-12 19:04:47,314 INFO [Listener at localhost/33505] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/java.io.tmpdir/Jetty_localhost_37035_datanode____odeget/webapp 2023-07-12 19:04:47,406 INFO [Listener at localhost/33505] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37035 2023-07-12 19:04:47,413 WARN [Listener at localhost/44411] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-12 19:04:47,507 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xef7870b3f45f7d42: Processing first storage report for DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd from datanode e7646b78-88d6-4cbc-ba92-654db454bb77 2023-07-12 19:04:47,507 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xef7870b3f45f7d42: from storage DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd node DatanodeRegistration(127.0.0.1:41995, datanodeUuid=e7646b78-88d6-4cbc-ba92-654db454bb77, infoPort=33385, infoSecurePort=0, ipcPort=44411, storageInfo=lv=-57;cid=testClusterID;nsid=1603840091;c=1689188687103), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-12 19:04:47,507 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xef7870b3f45f7d42: Processing first storage report for DS-63f31dfa-d7d9-4a09-9655-289f1c4c68ed from datanode e7646b78-88d6-4cbc-ba92-654db454bb77 2023-07-12 19:04:47,508 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xef7870b3f45f7d42: from storage DS-63f31dfa-d7d9-4a09-9655-289f1c4c68ed node DatanodeRegistration(127.0.0.1:41995, datanodeUuid=e7646b78-88d6-4cbc-ba92-654db454bb77, infoPort=33385, infoSecurePort=0, ipcPort=44411, storageInfo=lv=-57;cid=testClusterID;nsid=1603840091;c=1689188687103), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-12 19:04:47,519 DEBUG [Listener at localhost/44411] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac 2023-07-12 19:04:47,521 INFO [Listener at localhost/44411] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3/zookeeper_0, clientPort=63125, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-12 19:04:47,523 INFO [Listener at localhost/44411] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63125 2023-07-12 19:04:47,523 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,524 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,934 INFO [Listener at localhost/44411] util.FSUtils(471): Created version file at hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed with version=8 2023-07-12 19:04:47,934 INFO [Listener at localhost/44411] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:41637/user/jenkins/test-data/05e62127-0d34-cfa2-7a34-4f58e6a34aa3/hbase-staging 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:47,936 INFO [Listener at localhost/44411] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:47,937 INFO [Listener at localhost/44411] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42753 2023-07-12 19:04:47,937 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,938 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,939 INFO [Listener at localhost/44411] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42753 connecting to ZooKeeper ensemble=127.0.0.1:63125 2023-07-12 19:04:47,947 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:427530x0, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:47,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:42753-0x1015aec61a40000 connected 2023-07-12 19:04:47,960 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:47,961 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:47,961 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:47,962 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42753 2023-07-12 19:04:47,962 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42753 2023-07-12 19:04:47,962 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42753 2023-07-12 19:04:47,962 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42753 2023-07-12 19:04:47,962 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42753 2023-07-12 19:04:47,963 INFO [Listener at localhost/44411] master.HMaster(444): hbase.rootdir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed, hbase.cluster.distributed=false 2023-07-12 19:04:47,964 INFO [Listener at localhost/44411] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:47,964 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,964 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,964 INFO [Listener at localhost/44411] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:47,964 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:47,965 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:47,965 INFO [Listener at localhost/44411] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:47,965 INFO [Listener at localhost/44411] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45841 2023-07-12 19:04:47,966 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,966 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:47,967 INFO [Listener at localhost/44411] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45841 connecting to ZooKeeper ensemble=127.0.0.1:63125 2023-07-12 19:04:47,970 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:458410x0, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:47,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:45841-0x1015aec61a40001 connected 2023-07-12 19:04:47,973 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:47,973 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:47,974 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:47,974 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45841 2023-07-12 19:04:47,974 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45841 2023-07-12 19:04:47,979 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45841 2023-07-12 19:04:47,987 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45841 2023-07-12 19:04:47,988 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45841 2023-07-12 19:04:47,989 INFO [Listener at localhost/44411] master.HMaster(444): hbase.rootdir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed, hbase.cluster.distributed=false 2023-07-12 19:04:48,007 INFO [Listener at localhost/44411] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-12 19:04:48,007 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:48,007 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:48,008 INFO [Listener at localhost/44411] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-12 19:04:48,008 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-12 19:04:48,008 INFO [Listener at localhost/44411] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-12 19:04:48,008 INFO [Listener at localhost/44411] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-12 19:04:48,009 INFO [Listener at localhost/44411] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41561 2023-07-12 19:04:48,009 INFO [Listener at localhost/44411] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-12 19:04:48,015 DEBUG [Listener at localhost/44411] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-12 19:04:48,016 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:48,017 INFO [Listener at localhost/44411] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:48,018 INFO [Listener at localhost/44411] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41561 connecting to ZooKeeper ensemble=127.0.0.1:63125 2023-07-12 19:04:48,021 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:415610x0, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-12 19:04:48,022 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): regionserver:415610x0, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:48,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41561-0x1015aec61a40002 connected 2023-07-12 19:04:48,023 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:48,024 DEBUG [Listener at localhost/44411] zookeeper.ZKUtil(164): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-12 19:04:48,027 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41561 2023-07-12 19:04:48,027 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41561 2023-07-12 19:04:48,027 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41561 2023-07-12 19:04:48,033 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41561 2023-07-12 19:04:48,033 DEBUG [Listener at localhost/44411] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41561 2023-07-12 19:04:48,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:48,039 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,041 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:48,041 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:48,042 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,042 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:48,043 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:48,043 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:48,044 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,044 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,044 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:48,044 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45841,1689188687964 from backup master directory 2023-07-12 19:04:48,045 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:48,046 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:48,047 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,047 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:48,047 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(278): Another master is the active master, jenkins-hbase4.apache.org,45841,1689188687964; waiting to become the next active master 2023-07-12 19:04:48,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:48,047 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:48,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:48,047 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:48,048 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/hbase.id with ID: d33aa46a-b19a-491e-9003-7b5b4fc09ac4 2023-07-12 19:04:48,073 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:48,076 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,076 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4d44a721 to 127.0.0.1:63125 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:48,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d798f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:48,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:48,487 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:48,488 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:48,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store-tmp 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:48,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:48,896 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:48,896 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:48,897 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,899 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45841%2C1689188687964, suffix=, logDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964, archiveDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:48,913 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK] 2023-07-12 19:04:48,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:48,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK]] 2023-07-12 19:04:48,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:48,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:48,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,916 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,917 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,918 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:48,919 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:48,919 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:48,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,920 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,922 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:48,923 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:48,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59172739, jitterRate=-0.11825747787952423}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:48,924 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:48,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:48,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:48,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:48,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:48,925 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:48,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-07-12 19:04:48,926 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:48,926 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-12 19:04:48,927 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:48,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:48,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:48,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:48,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:48,938 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:48,940 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,940 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,940 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:48,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:48,941 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:48,942 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:48,942 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:48,943 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,943 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:48,943 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,45841,1689188687964, sessionid=0x1015aec61a40001, setting cluster-up flag (Was=false) 2023-07-12 19:04:48,945 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegionServer(951): ClusterId : d33aa46a-b19a-491e-9003-7b5b4fc09ac4 2023-07-12 19:04:48,948 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,948 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:48,958 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,960 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,960 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:48,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:48,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:48,967 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.hbase-snapshot/.tmp 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:48,969 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:48,969 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188718969 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:48,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:48,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:48,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:48,971 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:48,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:48,971 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-12 19:04:48,971 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:48,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188688972,5,FailOnTimeoutGroup] 2023-07-12 19:04:48,972 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188688972,5,FailOnTimeoutGroup] 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:48,972 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:48,972 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:49,037 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(951): ClusterId : d33aa46a-b19a-491e-9003-7b5b4fc09ac4 2023-07-12 19:04:49,037 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-12 19:04:49,039 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-12 19:04:49,039 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-12 19:04:49,040 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-12 19:04:49,041 DEBUG [RS:0;jenkins-hbase4:41561] zookeeper.ReadOnlyZKClient(139): Connect 0x2cea50c2 to 127.0.0.1:63125 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:49,050 DEBUG [RS:0;jenkins-hbase4:41561] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@222dab2f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:49,050 DEBUG [RS:0;jenkins-hbase4:41561] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26123bb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:49,059 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41561 2023-07-12 19:04:49,059 INFO [RS:0;jenkins-hbase4:41561] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-12 19:04:49,059 INFO [RS:0;jenkins-hbase4:41561] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-12 19:04:49,059 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1022): About to register with Master. 2023-07-12 19:04:49,059 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,45841,1689188687964 with isa=jenkins-hbase4.apache.org/172.31.14.131:41561, startcode=1689188688006 2023-07-12 19:04:49,059 DEBUG [RS:0;jenkins-hbase4:41561] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:49,062 INFO [RS-EventLoopGroup-16-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53493, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:49,063 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=45841] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,063 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed 2023-07-12 19:04:49,063 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:33505 2023-07-12 19:04:49,063 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-12 19:04:49,065 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:49,065 DEBUG [RS:0;jenkins-hbase4:41561] zookeeper.ZKUtil(162): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,066 WARN [RS:0;jenkins-hbase4:41561] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:49,066 INFO [RS:0;jenkins-hbase4:41561] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:49,066 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1948): logDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,066 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41561,1689188688006] 2023-07-12 19:04:49,069 DEBUG [RS:0;jenkins-hbase4:41561] zookeeper.ZKUtil(162): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,070 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-12 19:04:49,070 INFO [RS:0;jenkins-hbase4:41561] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-12 19:04:49,071 INFO [RS:0;jenkins-hbase4:41561] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-12 19:04:49,072 INFO [RS:0;jenkins-hbase4:41561] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-12 19:04:49,072 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,072 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-12 19:04:49,073 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,073 DEBUG [RS:0;jenkins-hbase4:41561] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:49,074 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,074 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,074 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,086 INFO [RS:0;jenkins-hbase4:41561] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-12 19:04:49,086 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41561,1689188688006-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:49,097 INFO [RS:0;jenkins-hbase4:41561] regionserver.Replication(203): jenkins-hbase4.apache.org,41561,1689188688006 started 2023-07-12 19:04:49,097 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41561,1689188688006, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41561, sessionid=0x1015aec61a40002 2023-07-12 19:04:49,097 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-12 19:04:49,097 DEBUG [RS:0;jenkins-hbase4:41561] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,097 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41561,1689188688006' 2023-07-12 19:04:49,097 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41561,1689188688006' 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-12 19:04:49,098 DEBUG [RS:0;jenkins-hbase4:41561] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-12 19:04:49,099 DEBUG [RS:0;jenkins-hbase4:41561] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-12 19:04:49,099 INFO [RS:0;jenkins-hbase4:41561] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-12 19:04:49,099 INFO [RS:0;jenkins-hbase4:41561] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-12 19:04:49,200 INFO [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41561%2C1689188688006, suffix=, logDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006, archiveDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/oldWALs, maxLogs=32 2023-07-12 19:04:49,215 DEBUG [RS-EventLoopGroup-17-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK] 2023-07-12 19:04:49,217 INFO [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006/jenkins-hbase4.apache.org%2C41561%2C1689188688006.1689188689201 2023-07-12 19:04:49,217 DEBUG [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK]] 2023-07-12 19:04:49,379 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:49,380 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:49,380 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed 2023-07-12 19:04:49,786 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:49,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:49,789 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/info 2023-07-12 19:04:49,789 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:49,790 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:49,790 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:49,791 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:49,791 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:49,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:49,792 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:49,793 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/table 2023-07-12 19:04:49,793 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:49,794 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:49,794 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740 2023-07-12 19:04:49,795 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740 2023-07-12 19:04:49,796 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:49,797 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:49,799 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70427123, jitterRate=0.04944591224193573}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:49,799 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:49,799 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:49,800 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:49,800 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:49,801 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:49,801 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-12 19:04:49,801 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-12 19:04:49,802 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-12 19:04:49,803 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-12 19:04:49,953 DEBUG [jenkins-hbase4:45841] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-12 19:04:49,954 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41561,1689188688006, state=OPENING 2023-07-12 19:04:49,955 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-12 19:04:49,957 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:49,957 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:49,957 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:49,957 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:49,958 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:50,110 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:50,110 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:50,112 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44492, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:50,115 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-12 19:04:50,116 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:50,117 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41561%2C1689188688006.meta, suffix=.meta, logDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006, archiveDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/oldWALs, maxLogs=32 2023-07-12 19:04:50,130 DEBUG [RS-EventLoopGroup-17-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK] 2023-07-12 19:04:50,132 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006/jenkins-hbase4.apache.org%2C41561%2C1689188688006.meta.1689188690117.meta 2023-07-12 19:04:50,132 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK]] 2023-07-12 19:04:50,132 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:50,132 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-12 19:04:50,133 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-12 19:04:50,133 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-12 19:04:50,133 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-12 19:04:50,133 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:50,133 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-12 19:04:50,133 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-12 19:04:50,134 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-12 19:04:50,135 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/info 2023-07-12 19:04:50,135 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/info 2023-07-12 19:04:50,135 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-12 19:04:50,136 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:50,136 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-12 19:04:50,137 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:50,137 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/rep_barrier 2023-07-12 19:04:50,137 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-12 19:04:50,138 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:50,138 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-12 19:04:50,139 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/table 2023-07-12 19:04:50,139 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/table 2023-07-12 19:04:50,139 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-12 19:04:50,139 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:50,140 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740 2023-07-12 19:04:50,141 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740 2023-07-12 19:04:50,142 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-12 19:04:50,143 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-12 19:04:50,144 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63747921, jitterRate=-0.05008195340633392}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-12 19:04:50,144 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-12 19:04:50,145 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689188690110 2023-07-12 19:04:50,147 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-12 19:04:50,148 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-12 19:04:50,148 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41561,1689188688006, state=OPEN 2023-07-12 19:04:50,152 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:50,152 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-12 19:04:50,152 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:50,152 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-12 19:04:50,154 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-12 19:04:50,154 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41561,1689188688006 in 195 msec 2023-07-12 19:04:50,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-12 19:04:50,158 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 353 msec 2023-07-12 19:04:50,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1910 sec 2023-07-12 19:04:50,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188690160, completionTime=-1 2023-07-12 19:04:50,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:50,160 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:50,163 DEBUG [hconnection-0x571b0a6d-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:50,165 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44508, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:50,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:50,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188750167 2023-07-12 19:04:50,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188810167 2023-07-12 19:04:50,167 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45841,1689188687964-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45841,1689188687964-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45841,1689188687964-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45841, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:50,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-12 19:04:50,175 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:50,175 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:50,176 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:50,177 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:50,177 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:50,179 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:50,180 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77 empty. 2023-07-12 19:04:50,180 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:50,180 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-12 19:04:50,588 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:50,589 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7e477b68840adb0e5568a48b431b3f77, NAME => 'hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 7e477b68840adb0e5568a48b431b3f77, disabling compactions & flushes 2023-07-12 19:04:50,995 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. after waiting 0 ms 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:50,995 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:50,995 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 7e477b68840adb0e5568a48b431b3f77: 2023-07-12 19:04:50,997 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:50,998 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188690998"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188690998"}]},"ts":"1689188690998"} 2023-07-12 19:04:51,001 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-12 19:04:51,001 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:51,001 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188691001"}]},"ts":"1689188691001"} 2023-07-12 19:04:51,003 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-12 19:04:51,005 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e477b68840adb0e5568a48b431b3f77, ASSIGN}] 2023-07-12 19:04:51,006 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=7e477b68840adb0e5568a48b431b3f77, ASSIGN 2023-07-12 19:04:51,007 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=7e477b68840adb0e5568a48b431b3f77, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=false 2023-07-12 19:04:51,158 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7e477b68840adb0e5568a48b431b3f77, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:51,158 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188691158"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188691158"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188691158"}]},"ts":"1689188691158"} 2023-07-12 19:04:51,160 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 7e477b68840adb0e5568a48b431b3f77, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:51,316 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:51,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7e477b68840adb0e5568a48b431b3f77, NAME => 'hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:51,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:51,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,317 INFO [StoreOpener-7e477b68840adb0e5568a48b431b3f77-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,319 DEBUG [StoreOpener-7e477b68840adb0e5568a48b431b3f77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/info 2023-07-12 19:04:51,319 DEBUG [StoreOpener-7e477b68840adb0e5568a48b431b3f77-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/info 2023-07-12 19:04:51,319 INFO [StoreOpener-7e477b68840adb0e5568a48b431b3f77-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e477b68840adb0e5568a48b431b3f77 columnFamilyName info 2023-07-12 19:04:51,319 INFO [StoreOpener-7e477b68840adb0e5568a48b431b3f77-1] regionserver.HStore(310): Store=7e477b68840adb0e5568a48b431b3f77/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:51,320 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,321 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,323 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:51,324 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:51,325 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7e477b68840adb0e5568a48b431b3f77; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64470529, jitterRate=-0.039314255118370056}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:51,325 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7e477b68840adb0e5568a48b431b3f77: 2023-07-12 19:04:51,326 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77., pid=6, masterSystemTime=1689188691312 2023-07-12 19:04:51,328 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:51,328 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:51,328 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=7e477b68840adb0e5568a48b431b3f77, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:51,328 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689188691328"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188691328"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188691328"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188691328"}]},"ts":"1689188691328"} 2023-07-12 19:04:51,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-12 19:04:51,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 7e477b68840adb0e5568a48b431b3f77, server=jenkins-hbase4.apache.org,41561,1689188688006 in 170 msec 2023-07-12 19:04:51,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-12 19:04:51,333 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=7e477b68840adb0e5568a48b431b3f77, ASSIGN in 326 msec 2023-07-12 19:04:51,333 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:51,333 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188691333"}]},"ts":"1689188691333"} 2023-07-12 19:04:51,334 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-12 19:04:51,339 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:51,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.1640 sec 2023-07-12 19:04:51,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-12 19:04:51,379 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:51,379 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:51,379 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:51,382 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:51,389 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:51,391 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 8 msec 2023-07-12 19:04:51,394 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:51,400 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-12 19:04:51,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 8 msec 2023-07-12 19:04:51,408 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:51,411 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:51,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.363sec 2023-07-12 19:04:51,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:51,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:51,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:51,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45841,1689188687964-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:51,412 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45841,1689188687964-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:51,413 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:51,438 DEBUG [Listener at localhost/44411] zookeeper.ReadOnlyZKClient(139): Connect 0x3a84d042 to 127.0.0.1:63125 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:51,443 DEBUG [Listener at localhost/44411] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71ecd22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:51,444 DEBUG [hconnection-0x1fb604e6-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:51,448 INFO [RS-EventLoopGroup-17-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52954, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:51,449 INFO [Listener at localhost/44411] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:51,449 WARN [Listener at localhost/44411] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-07-12 19:04:51,451 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:51,452 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_PRE_OPERATION 2023-07-12 19:04:51,453 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-12 19:04:51,457 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:51,457 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:51,457 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:51,457 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:51,457 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e empty. 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 empty. 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d empty. 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 empty. 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:51,458 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:51,458 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testEnableTableWithFailoverAtStep3 regions 2023-07-12 19:04:51,867 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp/data/default/testEnableTableWithFailoverAtStep3/.tabledesc/.tableinfo.0000000001 2023-07-12 19:04:51,869 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(7675): creating {ENCODED => 30230a0350610d4e317e202c64d538a0, NAME => 'testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.', STARTKEY => 'a', ENDKEY => 'b'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp 2023-07-12 19:04:51,869 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(7675): creating {ENCODED => fdca952efa3dd0e15362758cf1ed720d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.', STARTKEY => 'b', ENDKEY => 'c'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp 2023-07-12 19:04:51,869 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5a9c075248bfea33c3c629a9f775522e, NAME => 'testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.', STARTKEY => '', ENDKEY => 'a'}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing 5a9c075248bfea33c3c629a9f775522e, disabling compactions & flushes 2023-07-12 19:04:51,877 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. after waiting 0 ms 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:51,877 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:51,877 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for 5a9c075248bfea33c3c629a9f775522e: 2023-07-12 19:04:51,877 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(7675): creating {ENCODED => b8c2da22e126648557cf9d684e1042b7, NAME => 'testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.', STARTKEY => 'c', ENDKEY => ''}, tableDescriptor='testEnableTableWithFailoverAtStep3', {NAME => 'f1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'f2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.tmp 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1604): Closing 30230a0350610d4e317e202c64d538a0, disabling compactions & flushes 2023-07-12 19:04:52,276 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. after waiting 0 ms 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1604): Closing fdca952efa3dd0e15362758cf1ed720d, disabling compactions & flushes 2023-07-12 19:04:52,276 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,276 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,276 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-1] regionserver.HRegion(1558): Region close journal for 30230a0350610d4e317e202c64d538a0: 2023-07-12 19:04:52,277 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,277 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. after waiting 0 ms 2023-07-12 19:04:52,277 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,277 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,277 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-2] regionserver.HRegion(1558): Region close journal for fdca952efa3dd0e15362758cf1ed720d: 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1604): Closing b8c2da22e126648557cf9d684e1042b7, disabling compactions & flushes 2023-07-12 19:04:52,283 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. after waiting 0 ms 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,283 INFO [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,283 DEBUG [RegionOpenAndInit-testEnableTableWithFailoverAtStep3-pool-0] regionserver.HRegion(1558): Region close journal for b8c2da22e126648557cf9d684e1042b7: 2023-07-12 19:04:52,285 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ADD_TO_META 2023-07-12 19:04:52,286 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692286"}]},"ts":"1689188692286"} 2023-07-12 19:04:52,286 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692286"}]},"ts":"1689188692286"} 2023-07-12 19:04:52,286 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692286"}]},"ts":"1689188692286"} 2023-07-12 19:04:52,286 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692286"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692286"}]},"ts":"1689188692286"} 2023-07-12 19:04:52,290 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 4 regions to meta. 2023-07-12 19:04:52,291 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-12 19:04:52,291 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188692291"}]},"ts":"1689188692291"} 2023-07-12 19:04:52,292 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:52,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN}] 2023-07-12 19:04:52,297 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN 2023-07-12 19:04:52,297 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=false 2023-07-12 19:04:52,298 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN 2023-07-12 19:04:52,299 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=false 2023-07-12 19:04:52,299 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN 2023-07-12 19:04:52,300 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=false 2023-07-12 19:04:52,300 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN 2023-07-12 19:04:52,301 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=false 2023-07-12 19:04:52,449 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692449"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692449"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692449"}]},"ts":"1689188692449"} 2023-07-12 19:04:52,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=12, state=RUNNABLE; OpenRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,455 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,455 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692455"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692455"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692455"}]},"ts":"1689188692455"} 2023-07-12 19:04:52,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=11, state=RUNNABLE; OpenRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,461 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,461 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692461"}]},"ts":"1689188692461"} 2023-07-12 19:04:52,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=10, state=RUNNABLE; OpenRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,466 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,466 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692466"}]},"ts":"1689188692466"} 2023-07-12 19:04:52,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=13, state=RUNNABLE; OpenRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fdca952efa3dd0e15362758cf1ed720d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 19:04:52,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,609 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,610 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,611 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f1 2023-07-12 19:04:52,611 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f1 2023-07-12 19:04:52,611 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fdca952efa3dd0e15362758cf1ed720d columnFamilyName f1 2023-07-12 19:04:52,612 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(310): Store=fdca952efa3dd0e15362758cf1ed720d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,612 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,613 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f2 2023-07-12 19:04:52,613 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f2 2023-07-12 19:04:52,613 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fdca952efa3dd0e15362758cf1ed720d columnFamilyName f2 2023-07-12 19:04:52,613 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(310): Store=fdca952efa3dd0e15362758cf1ed720d/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,614 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:52,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:52,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fdca952efa3dd0e15362758cf1ed720d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63401957, jitterRate=-0.055237218737602234}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:52,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fdca952efa3dd0e15362758cf1ed720d: 2023-07-12 19:04:52,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d., pid=14, masterSystemTime=1689188692604 2023-07-12 19:04:52,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,621 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8c2da22e126648557cf9d684e1042b7, NAME => 'testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 19:04:52,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,622 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692622"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188692622"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188692622"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188692622"}]},"ts":"1689188692622"} 2023-07-12 19:04:52,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,623 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,624 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f1 2023-07-12 19:04:52,625 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f1 2023-07-12 19:04:52,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=12 2023-07-12 19:04:52,625 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8c2da22e126648557cf9d684e1042b7 columnFamilyName f1 2023-07-12 19:04:52,625 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=12, state=SUCCESS; OpenRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006 in 171 msec 2023-07-12 19:04:52,625 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(310): Store=b8c2da22e126648557cf9d684e1042b7/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,625 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN in 330 msec 2023-07-12 19:04:52,627 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f2 2023-07-12 19:04:52,627 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f2 2023-07-12 19:04:52,627 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8c2da22e126648557cf9d684e1042b7 columnFamilyName f2 2023-07-12 19:04:52,627 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(310): Store=b8c2da22e126648557cf9d684e1042b7/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,628 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,629 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:52,630 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:52,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b8c2da22e126648557cf9d684e1042b7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73068114, jitterRate=0.08879974484443665}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:52,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b8c2da22e126648557cf9d684e1042b7: 2023-07-12 19:04:52,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7., pid=17, masterSystemTime=1689188692604 2023-07-12 19:04:52,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 30230a0350610d4e317e202c64d538a0, NAME => 'testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 19:04:52,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,635 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,636 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692635"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188692635"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188692635"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188692635"}]},"ts":"1689188692635"} 2023-07-12 19:04:52,637 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,638 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f1 2023-07-12 19:04:52,639 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f1 2023-07-12 19:04:52,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=13 2023-07-12 19:04:52,639 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 30230a0350610d4e317e202c64d538a0 columnFamilyName f1 2023-07-12 19:04:52,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=13, state=SUCCESS; OpenRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006 in 169 msec 2023-07-12 19:04:52,640 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(310): Store=30230a0350610d4e317e202c64d538a0/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,640 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN in 344 msec 2023-07-12 19:04:52,641 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f2 2023-07-12 19:04:52,641 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f2 2023-07-12 19:04:52,642 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 30230a0350610d4e317e202c64d538a0 columnFamilyName f2 2023-07-12 19:04:52,642 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(310): Store=30230a0350610d4e317e202c64d538a0/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,645 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:52,646 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:52,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 30230a0350610d4e317e202c64d538a0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61595605, jitterRate=-0.08215396106243134}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:52,649 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 30230a0350610d4e317e202c64d538a0: 2023-07-12 19:04:52,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0., pid=15, masterSystemTime=1689188692604 2023-07-12 19:04:52,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5a9c075248bfea33c3c629a9f775522e, NAME => 'testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 19:04:52,652 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,652 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692652"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188692652"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188692652"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188692652"}]},"ts":"1689188692652"} 2023-07-12 19:04:52,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:52,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,653 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=11 2023-07-12 19:04:52,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=11, state=SUCCESS; OpenRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006 in 196 msec 2023-07-12 19:04:52,655 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f1 2023-07-12 19:04:52,655 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f1 2023-07-12 19:04:52,655 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a9c075248bfea33c3c629a9f775522e columnFamilyName f1 2023-07-12 19:04:52,656 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN in 360 msec 2023-07-12 19:04:52,656 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(310): Store=5a9c075248bfea33c3c629a9f775522e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,656 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,657 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f2 2023-07-12 19:04:52,657 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f2 2023-07-12 19:04:52,658 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a9c075248bfea33c3c629a9f775522e columnFamilyName f2 2023-07-12 19:04:52,658 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(310): Store=5a9c075248bfea33c3c629a9f775522e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:52,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,662 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:52,663 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-12 19:04:52,665 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5a9c075248bfea33c3c629a9f775522e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63100126, jitterRate=-0.059734851121902466}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:52,665 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5a9c075248bfea33c3c629a9f775522e: 2023-07-12 19:04:52,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e., pid=16, masterSystemTime=1689188692604 2023-07-12 19:04:52,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,668 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,668 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692668"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188692668"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188692668"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188692668"}]},"ts":"1689188692668"} 2023-07-12 19:04:52,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=10 2023-07-12 19:04:52,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=10, state=SUCCESS; OpenRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006 in 207 msec 2023-07-12 19:04:52,672 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-12 19:04:52,673 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN in 376 msec 2023-07-12 19:04:52,673 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-12 19:04:52,673 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188692673"}]},"ts":"1689188692673"} 2023-07-12 19:04:52,674 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 19:04:52,677 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 execute state=CREATE_TABLE_POST_OPERATION 2023-07-12 19:04:52,678 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 in 1.2270 sec 2023-07-12 19:04:52,704 DEBUG [Listener at localhost/44411] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-12 19:04:52,706 INFO [RS-EventLoopGroup-16-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-12 19:04:52,706 INFO [Listener at localhost/44411] client.HBaseAdmin$15(890): Started disable of testEnableTableWithFailoverAtStep3 2023-07-12 19:04:52,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45841] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testEnableTableWithFailoverAtStep3 2023-07-12 19:04:52,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45841] procedure2.ProcedureExecutor(1029): Stored pid=18, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:52,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45841] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:52,710 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188692710"}]},"ts":"1689188692710"} 2023-07-12 19:04:52,712 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLING in hbase:meta 2023-07-12 19:04:52,714 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testEnableTableWithFailoverAtStep3 to state=DISABLING 2023-07-12 19:04:52,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, UNASSIGN}, {pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, UNASSIGN}, {pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, UNASSIGN}, {pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, UNASSIGN}] 2023-07-12 19:04:52,715 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, UNASSIGN 2023-07-12 19:04:52,716 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,716 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692716"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692716"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692716"}]},"ts":"1689188692716"} 2023-07-12 19:04:52,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; CloseRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,719 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, UNASSIGN 2023-07-12 19:04:52,720 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,720 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692720"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692720"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692720"}]},"ts":"1689188692720"} 2023-07-12 19:04:52,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=21, state=RUNNABLE; CloseRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,723 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, UNASSIGN 2023-07-12 19:04:52,724 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,724 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692724"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692724"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692724"}]},"ts":"1689188692724"} 2023-07-12 19:04:52,725 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=20, state=RUNNABLE; CloseRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,727 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=18, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, UNASSIGN 2023-07-12 19:04:52,727 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:52,727 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692727"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188692727"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188692727"}]},"ts":"1689188692727"} 2023-07-12 19:04:52,728 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE; CloseRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:52,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45841] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:52,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5a9c075248bfea33c3c629a9f775522e, disabling compactions & flushes 2023-07-12 19:04:52,870 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. after waiting 0 ms 2023-07-12 19:04:52,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,874 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:52,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:52,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5a9c075248bfea33c3c629a9f775522e: 2023-07-12 19:04:52,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:52,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b8c2da22e126648557cf9d684e1042b7, disabling compactions & flushes 2023-07-12 19:04:52,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. after waiting 0 ms 2023-07-12 19:04:52,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSED 2023-07-12 19:04:52,877 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692877"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692877"}]},"ts":"1689188692877"} 2023-07-12 19:04:52,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=19 2023-07-12 19:04:52,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:52,881 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; CloseRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006 in 151 msec 2023-07-12 19:04:52,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:52,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b8c2da22e126648557cf9d684e1042b7: 2023-07-12 19:04:52,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, UNASSIGN in 166 msec 2023-07-12 19:04:52,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:52,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 30230a0350610d4e317e202c64d538a0, disabling compactions & flushes 2023-07-12 19:04:52,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. after waiting 0 ms 2023-07-12 19:04:52,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,883 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=22 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSED 2023-07-12 19:04:52,884 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188692883"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692883"}]},"ts":"1689188692883"} 2023-07-12 19:04:52,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-12 19:04:52,886 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; CloseRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006 in 168 msec 2023-07-12 19:04:52,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:52,887 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, UNASSIGN in 171 msec 2023-07-12 19:04:52,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:52,888 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 30230a0350610d4e317e202c64d538a0: 2023-07-12 19:04:52,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:52,889 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(111): Close fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fdca952efa3dd0e15362758cf1ed720d, disabling compactions & flushes 2023-07-12 19:04:52,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,890 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=CLOSED 2023-07-12 19:04:52,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. after waiting 0 ms 2023-07-12 19:04:52,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,890 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692890"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692890"}]},"ts":"1689188692890"} 2023-07-12 19:04:52,893 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=20 2023-07-12 19:04:52,893 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=20, state=SUCCESS; CloseRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006 in 166 msec 2023-07-12 19:04:52,893 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-07-12 19:04:52,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, UNASSIGN in 178 msec 2023-07-12 19:04:52,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:52,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fdca952efa3dd0e15362758cf1ed720d: 2023-07-12 19:04:52,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(149): Closed fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:52,896 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSED 2023-07-12 19:04:52,896 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188692896"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689188692896"}]},"ts":"1689188692896"} 2023-07-12 19:04:52,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=21 2023-07-12 19:04:52,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=21, state=SUCCESS; CloseRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006 in 176 msec 2023-07-12 19:04:52,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=18 2023-07-12 19:04:52,902 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=18, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, UNASSIGN in 185 msec 2023-07-12 19:04:52,903 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188692903"}]},"ts":"1689188692903"} 2023-07-12 19:04:52,904 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=DISABLED in hbase:meta 2023-07-12 19:04:52,906 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testEnableTableWithFailoverAtStep3 to state=DISABLED 2023-07-12 19:04:52,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 in 200 msec 2023-07-12 19:04:53,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45841] master.MasterRpcServices(1230): Checking to see if procedure is done pid=18 2023-07-12 19:04:53,014 INFO [Listener at localhost/44411] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testEnableTableWithFailoverAtStep3, procId: 18 completed 2023-07-12 19:04:53,014 WARN [Listener at localhost/44411] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-07-12 19:04:53,019 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,021 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:53,021 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,021 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:53,021 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:53,269 INFO [Listener at localhost/44411] procedure.TestMasterFailoverWithProcedures(306): Restart 0 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,269 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:53,269 INFO [Listener at localhost/44411] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:53,269 INFO [Listener at localhost/44411] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:53,271 ERROR [Listener at localhost/44411] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:53,271 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:53,271 INFO [Listener at localhost/44411] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:53,272 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:53,272 INFO [Listener at localhost/44411] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:53,272 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:53,273 DEBUG [Listener at localhost/44411] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[] 2023-07-12 19:04:53,273 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:53,274 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:53,274 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:53,274 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:53,274 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,275 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,275 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,275 DEBUG [Listener at localhost/44411] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:53,275 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:53,276 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:53,276 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,41561,1689188688006, table=hbase:meta, region=1588230740 2023-07-12 19:04:53,276 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188723276 2023-07-12 19:04:53,277 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:53,277 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,279 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=7e477b68840adb0e5568a48b431b3f77, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,280 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,280 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=30230a0350610d4e317e202c64d538a0, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,280 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,280 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,280 INFO [Listener at localhost/44411] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:53,281 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188753281 2023-07-12 19:04:53,281 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188813281 2023-07-12 19:04:53,281 INFO [Listener at localhost/44411] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:53,281 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:53,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:53,282 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:53,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] procedure.TestMasterFailoverWithProcedures(306): Restart 1 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:53,531 ERROR [Listener at localhost/44411] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:53,531 INFO [Listener at localhost/44411] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:53,532 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:53,532 INFO [Listener at localhost/44411] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:53,532 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:53,533 DEBUG [Listener at localhost/44411] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0] 2023-07-12 19:04:53,533 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:53,534 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:53,534 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:53,534 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:53,535 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,535 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,536 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,536 DEBUG [Listener at localhost/44411] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:53,536 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 3 msec 2023-07-12 19:04:53,536 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:53,537 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,41561,1689188688006, table=hbase:meta, region=1588230740 2023-07-12 19:04:53,537 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188723537 2023-07-12 19:04:53,539 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:53,539 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,541 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=7e477b68840adb0e5568a48b431b3f77, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,541 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,541 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=30230a0350610d4e317e202c64d538a0, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,542 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,542 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,542 INFO [Listener at localhost/44411] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:53,542 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188753542 2023-07-12 19:04:53,542 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188813542 2023-07-12 19:04:53,542 INFO [Listener at localhost/44411] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-07-12 19:04:53,542 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:53,544 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,544 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188693544"}]},"ts":"1689188693544"} 2023-07-12 19:04:53,545 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:53,547 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:53,547 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:53,547 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:53,792 INFO [Listener at localhost/44411] procedure.TestMasterFailoverWithProcedures(306): Restart 2 exec state: pid=27, state=RUNNABLE:ENABLE_TABLE_PRE_OPERATION, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,792 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-07-12 19:04:53,792 INFO [Listener at localhost/44411] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:53,792 INFO [Listener at localhost/44411] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:53,793 ERROR [Listener at localhost/44411] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:53,793 INFO [Listener at localhost/44411] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-07-12 19:04:53,793 INFO [Listener at localhost/44411] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:53,793 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:53,793 INFO [Listener at localhost/44411] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:53,794 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:53,794 DEBUG [Listener at localhost/44411] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1] 2023-07-12 19:04:53,795 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:53,795 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:53,795 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:53,795 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:53,796 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,796 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,796 DEBUG [Listener at localhost/44411] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,796 DEBUG [Listener at localhost/44411] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:53,796 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 2 msec 2023-07-12 19:04:53,797 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:53,797 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,41561,1689188688006, table=hbase:meta, region=1588230740 2023-07-12 19:04:53,797 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188723797 2023-07-12 19:04:53,799 DEBUG [Listener at localhost/44411] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:53,799 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_SET_ENABLING_TABLE_STATE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,801 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=7e477b68840adb0e5568a48b431b3f77, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,801 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,801 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=30230a0350610d4e317e202c64d538a0, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,801 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,801 DEBUG [Listener at localhost/44411] assignment.RegionStateStore(147): Load hbase:meta entry region=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:53,801 INFO [Listener at localhost/44411] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:53,801 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188753801 2023-07-12 19:04:53,801 INFO [Listener at localhost/44411] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188813801 2023-07-12 19:04:53,802 INFO [Listener at localhost/44411] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-07-12 19:04:53,802 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,802 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188693802"}]},"ts":"1689188693802"} 2023-07-12 19:04:53,803 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLING in hbase:meta 2023-07-12 19:04:53,805 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-07-12 19:04:53,807 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 19:04:53,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN}] 2023-07-12 19:04:53,807 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-07-12 19:04:53,807 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:53,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:53,807 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=27, state=WAITING:ENABLE_TABLE_SET_ENABLED_TABLE_STATE, locked=true; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-07-12 19:04:54,052 INFO [Listener at localhost/44411] procedure.TestMasterFailoverWithProcedures(313): Trigger master failover 2023-07-12 19:04:54,052 INFO [Listener at localhost/44411] hbase.MiniHBaseCluster(607): Aborting jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:54,052 ERROR [Listener at localhost/44411] helpers.MarkerIgnoringBase(143): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-12 19:04:54,052 ERROR [Listener at localhost/44411] helpers.MarkerIgnoringBase(159): ***** ABORTING master jenkins-hbase4.apache.org,45841,1689188687964: Aborting for tests ***** java.lang.Exception: Trace info at org.apache.hadoop.hbase.MiniHBaseCluster.abortMaster(MiniHBaseCluster.java:608) at org.apache.hadoop.hbase.MiniHBaseCluster.killMaster(MiniHBaseCluster.java:411) at org.apache.hadoop.hbase.master.procedure.MasterProcedureTestingUtility.masterFailover(MasterProcedureTestingUtility.java:135) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testRecoveryAndDoubleExecution(TestMasterFailoverWithProcedures.java:314) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailoverAtStep(TestMasterFailoverWithProcedures.java:289) at org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures.testEnableTableWithFailover(TestMasterFailoverWithProcedures.java:270) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:54,052 INFO [Listener at localhost/44411] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,45841,1689188687964' ***** 2023-07-12 19:04:54,052 INFO [Listener at localhost/44411] regionserver.HRegionServer(2311): STOPPED: Stopped by Listener at localhost/44411 2023-07-12 19:04:54,053 DEBUG [M:1;jenkins-hbase4:45841] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71e5a75c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:54,053 INFO [M:1;jenkins-hbase4:45841] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,45841,1689188687964 2023-07-12 19:04:54,056 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,056 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,056 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:54,056 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,056 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:54,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:54,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:54,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:54,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:54,057 DEBUG [zk-event-processor-pool-0] master.ActiveMasterManager(186): No master available. Notifying waiting threads 2023-07-12 19:04:54,057 DEBUG [M:1;jenkins-hbase4:45841] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d44a721 to 127.0.0.1:63125 2023-07-12 19:04:54,057 DEBUG [M:1;jenkins-hbase4:45841] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:54,058 INFO [M:1;jenkins-hbase4:45841] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,45841,1689188687964; all regions closed. 2023-07-12 19:04:54,058 DEBUG [M:1;jenkins-hbase4:45841] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:54,058 DEBUG [M:1;jenkins-hbase4:45841] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:54,058 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:54,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188688972] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188688972,5,FailOnTimeoutGroup] 2023-07-12 19:04:54,058 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188688972] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188688972,5,FailOnTimeoutGroup] 2023-07-12 19:04:54,058 DEBUG [M:1;jenkins-hbase4:45841] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:54,058 INFO [M:1;jenkins-hbase4:45841] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:54,058 INFO [M:1;jenkins-hbase4:45841] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:54,058 INFO [M:1;jenkins-hbase4:45841] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:54,059 DEBUG [M:1;jenkins-hbase4:45841] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:54,059 INFO [M:1;jenkins-hbase4:45841] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:54,059 ERROR [M:1;jenkins-hbase4:45841] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:54,059 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,059 INFO [M:1;jenkins-hbase4:45841] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=true 2023-07-12 19:04:54,059 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:54,059 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:54,060 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,060 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:54,060 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:54,061 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:54,061 INFO [M:1;jenkins-hbase4:45841] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:54,062 INFO [M:1;jenkins-hbase4:45841] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2023-07-12 19:04:54,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42753,1689188687935 from backup master directory 2023-07-12 19:04:54,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-12 19:04:54,064 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:54,064 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:54,064 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-12 19:04:54,064 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-12 19:04:54,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:54,067 DEBUG [M:1;jenkins-hbase4:45841] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:54,067 INFO [M:1;jenkins-hbase4:45841] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:54,067 DEBUG [M:1;jenkins-hbase4:45841] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:54,067 DEBUG [M:1;jenkins-hbase4:45841] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:54,067 DEBUG [M:1;jenkins-hbase4:45841] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:54,067 ERROR [M:1;jenkins-hbase4:45841] regionserver.HRegion(1825): Memstore data size is 89368 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:54,067 INFO [M:1;jenkins-hbase4:45841] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:54,067 DEBUG [M:1;jenkins-hbase4:45841] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:54,067 INFO [M:1;jenkins-hbase4:45841] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:54,068 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:54,068 INFO [M:1;jenkins-hbase4:45841] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45841 2023-07-12 19:04:54,072 DEBUG [M:1;jenkins-hbase4:45841] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45841,1689188687964 already deleted, retry=false 2023-07-12 19:04:54,075 WARN [RS-EventLoopGroup-17-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:45841 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45841 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-12 19:04:54,075 DEBUG [RS-EventLoopGroup-17-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:45841 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:45841 2023-07-12 19:04:54,083 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-12 19:04:54,176 DEBUG [RS:0;jenkins-hbase4:41561] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-12 19:04:54,178 INFO [RS-EventLoopGroup-15-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56161, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-07-12 19:04:54,178 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,260 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:54,260 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:45841-0x1015aec61a40001, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:54,260 INFO [M:1;jenkins-hbase4:45841] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,45841,1689188687964; zookeeper connection closed. 2023-07-12 19:04:54,280 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,382 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,484 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,510 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x14d6b724 to 127.0.0.1:63125 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-12 19:04:54,514 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@149f9b3b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-12 19:04:54,514 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-12 19:04:54,515 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-12 19:04:54,515 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-12 19:04:54,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(288): Renamed hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964 to hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964-dead as it is dead 2023-07-12 19:04:54,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964-dead/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:54,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964-dead/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 after 0ms 2023-07-12 19:04:54,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(300): Renamed hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964-dead/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 to hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:54,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(302): Delete empty local region wal dir hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,45841,1689188687964-dead 2023-07-12 19:04:54,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:54,527 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42753%2C1689188687935, suffix=, logDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,42753,1689188687935, archiveDir=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/oldWALs, maxLogs=10 2023-07-12 19:04:54,545 DEBUG [RS-EventLoopGroup-17-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK] 2023-07-12 19:04:54,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/WALs/jenkins-hbase4.apache.org,42753,1689188687935/jenkins-hbase4.apache.org%2C42753%2C1689188687935.1689188694527 2023-07-12 19:04:54,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41995,DS-23ce703a-6dfe-4c7b-be5a-0e33a0dca1dd,DISK]] 2023-07-12 19:04:54,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-12 19:04:54,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:54,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:54,551 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:54,552 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:54,553 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-12 19:04:54,554 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-12 19:04:54,555 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:54,555 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5179): Found 1 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals 2023-07-12 19:04:54,555 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:54,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5464): Applied 231, skipped 0, firstSequenceIdInLog=3, maxSequenceIdInLog=193, path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:54,563 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=87.27 KB heapSize=104.57 KB 2023-07-12 19:04:54,586 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,687 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,789 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,891 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,975 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=87.27 KB at sequenceid=193 (bloomFilter=true), to=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7e559e6d8fdd44df87137d56faa20345 2023-07-12 19:04:54,981 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7e559e6d8fdd44df87137d56faa20345 as hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7e559e6d8fdd44df87137d56faa20345 2023-07-12 19:04:54,985 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HStore(1080): Added hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7e559e6d8fdd44df87137d56faa20345, entries=231, sequenceid=193, filesize=92.5 K 2023-07-12 19:04:54,986 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(2948): Finished flush of dataSize ~87.27 KB/89368, heapSize ~104.55 KB/107064, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 422ms, sequenceid=193, compaction requested=false; wal=null 2023-07-12 19:04:54,986 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/jenkins-hbase4.apache.org%2C45841%2C1689188687964.1689188688900 2023-07-12 19:04:54,989 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-12 19:04:54,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/193.seqid, newMaxSeqId=193, maxSeqId=1 2023-07-12 19:04:54,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=194; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64631004, jitterRate=-0.036922991275787354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-12 19:04:54,992 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:54,992 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-12 19:04:54,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-07-12 19:04:54,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-07-12 19:04:54,993 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42753] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:546) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15954) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-12 19:04:54,994 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-12 19:04:54,994 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-07-12 19:04:54,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] store.ProcedureTree(114): Procedure Procedure(pid=27, ppid=-1, class=org.apache.hadoop.hbase.master.procedure.EnableTableProcedure) stack ids=[0, 1, 2] 2023-07-12 19:04:54,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-07-12 19:04:54,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-07-12 19:04:54,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-07-12 19:04:54,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-07-12 19:04:54,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:54,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(411): Completed pid=18, state=SUCCESS; DisableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:54,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(468): Loading pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:54,999 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.Procedure(966): pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 held the lock before restarting, call acquireLock to restore it. 2023-07-12 19:04:54,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-07-12 19:04:54,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-12 19:04:55,000 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-07-12 19:04:55,000 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,41561,1689188688006, table=hbase:meta, region=1588230740 2023-07-12 19:04:55,001 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2023-07-12 19:04:55,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,012 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-12 19:04:55,013 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-12 19:04:55,013 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-12 19:04:55,013 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-12 19:04:55,013 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-12 19:04:55,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-12 19:04:55,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-12 19:04:55,015 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-12 19:04:55,015 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42753,1689188687935, sessionid=0x1015aec61a40000, setting cluster-up flag (Was=true) 2023-07-12 19:04:55,017 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-12 19:04:55,018 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:55,019 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-12 19:04:55,020 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:55,021 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/.hbase-snapshot/.tmp 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-12 19:04:55,021 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-12 19:04:55,022 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689188725022 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-12 19:04:55,023 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=27, state=RUNNABLE:ENABLE_TABLE_MARK_REGIONS_ONLINE; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,023 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-12 19:04:55,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188695024,5,FailOnTimeoutGroup] 2023-07-12 19:04:55,024 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188695024,5,FailOnTimeoutGroup] 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,024 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-12 19:04:55,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689188695025, completionTime=-1 2023-07-12 19:04:55,025 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-07-12 19:04:55,025 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-12 19:04:55,026 DEBUG [hconnection-0x5cf82958-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-12 19:04:55,027 INFO [RS-EventLoopGroup-17-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52964, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-12 19:04:55,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=7e477b68840adb0e5568a48b431b3f77, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:55,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=5a9c075248bfea33c3c629a9f775522e, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:55,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=30230a0350610d4e317e202c64d538a0, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:55,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=fdca952efa3dd0e15362758cf1ed720d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:55,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.RegionStateStore(147): Load hbase:meta entry region=b8c2da22e126648557cf9d684e1042b7, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,41561,1689188688006, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006, openSeqNum=2 2023-07-12 19:04:55,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-07-12 19:04:55,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689188755029 2023-07-12 19:04:55,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689188815029 2023-07-12 19:04:55,029 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-07-12 19:04:55,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42753,1689188687935-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42753,1689188687935-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42753,1689188687935-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42753, period=300000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,040 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-12 19:04:55,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on existing znode=/hbase/namespace 2023-07-12 19:04:55,042 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-12 19:04:55,049 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-12 19:04:55,055 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-12 19:04:55,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 0.991sec 2023-07-12 19:04:55,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-12 19:04:55,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-12 19:04:55,055 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-07-12 19:04:55,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-12 19:04:55,055 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42753,1689188687935-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-12 19:04:55,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN}, {pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN}, {pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN}, {pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN}] 2023-07-12 19:04:55,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42753,1689188687935-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-12 19:04:55,060 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN 2023-07-12 19:04:55,061 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,061 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=true 2023-07-12 19:04:55,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-12 19:04:55,062 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN 2023-07-12 19:04:55,062 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,062 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=true 2023-07-12 19:04:55,063 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN 2023-07-12 19:04:55,063 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,064 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=true 2023-07-12 19:04:55,064 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN 2023-07-12 19:04:55,065 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,065 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=28, ppid=27, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,41561,1689188688006; forceNewPlan=false, retain=true 2023-07-12 19:04:55,070 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-12 19:04:55,071 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-12 19:04:55,213 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,213 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188695213"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188695213"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188695213"}]},"ts":"1689188695213"} 2023-07-12 19:04:55,215 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:55,217 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,217 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188695217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188695217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188695217"}]},"ts":"1689188695217"} 2023-07-12 19:04:55,218 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=29, state=RUNNABLE; OpenRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:55,220 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,220 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188695220"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188695220"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188695220"}]},"ts":"1689188695220"} 2023-07-12 19:04:55,222 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=28, state=RUNNABLE; OpenRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:55,224 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,224 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188695224"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689188695224"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689188695224"}]},"ts":"1689188695224"} 2023-07-12 19:04:55,225 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=35, ppid=31, state=RUNNABLE; OpenRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006}] 2023-07-12 19:04:55,367 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,367 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-12 19:04:55,369 INFO [RS-EventLoopGroup-17-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52970, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-12 19:04:55,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fdca952efa3dd0e15362758cf1ed720d, NAME => 'testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.', STARTKEY => 'b', ENDKEY => 'c'} 2023-07-12 19:04:55,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:55,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,373 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,374 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f1 2023-07-12 19:04:55,374 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f1 2023-07-12 19:04:55,374 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fdca952efa3dd0e15362758cf1ed720d columnFamilyName f1 2023-07-12 19:04:55,375 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(310): Store=fdca952efa3dd0e15362758cf1ed720d/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,375 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,376 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f2 2023-07-12 19:04:55,376 DEBUG [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/f2 2023-07-12 19:04:55,376 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fdca952efa3dd0e15362758cf1ed720d columnFamilyName f2 2023-07-12 19:04:55,376 INFO [StoreOpener-fdca952efa3dd0e15362758cf1ed720d-1] regionserver.HStore(310): Store=fdca952efa3dd0e15362758cf1ed720d/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:55,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fdca952efa3dd0e15362758cf1ed720d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70146174, jitterRate=0.045259445905685425}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:55,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fdca952efa3dd0e15362758cf1ed720d: 2023-07-12 19:04:55,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d., pid=32, masterSystemTime=1689188695367 2023-07-12 19:04:55,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:55,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8c2da22e126648557cf9d684e1042b7, NAME => 'testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.', STARTKEY => 'c', ENDKEY => ''} 2023-07-12 19:04:55,384 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=fdca952efa3dd0e15362758cf1ed720d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,384 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188695384"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188695384"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188695384"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188695384"}]},"ts":"1689188695384"} 2023-07-12 19:04:55,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:55,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,386 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,387 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f1 2023-07-12 19:04:55,387 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f1 2023-07-12 19:04:55,387 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8c2da22e126648557cf9d684e1042b7 columnFamilyName f1 2023-07-12 19:04:55,388 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(310): Store=b8c2da22e126648557cf9d684e1042b7/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-12 19:04:55,388 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure fdca952efa3dd0e15362758cf1ed720d, server=jenkins-hbase4.apache.org,41561,1689188688006 in 171 msec 2023-07-12 19:04:55,389 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f2 2023-07-12 19:04:55,389 DEBUG [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/f2 2023-07-12 19:04:55,389 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=fdca952efa3dd0e15362758cf1ed720d, ASSIGN in 333 msec 2023-07-12 19:04:55,389 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8c2da22e126648557cf9d684e1042b7 columnFamilyName f2 2023-07-12 19:04:55,390 INFO [StoreOpener-b8c2da22e126648557cf9d684e1042b7-1] regionserver.HStore(310): Store=b8c2da22e126648557cf9d684e1042b7/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:55,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b8c2da22e126648557cf9d684e1042b7; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64560430, jitterRate=-0.03797462582588196}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:55,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b8c2da22e126648557cf9d684e1042b7: 2023-07-12 19:04:55,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7., pid=35, masterSystemTime=1689188695367 2023-07-12 19:04:55,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:55,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:55,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 30230a0350610d4e317e202c64d538a0, NAME => 'testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.', STARTKEY => 'a', ENDKEY => 'b'} 2023-07-12 19:04:55,398 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=b8c2da22e126648557cf9d684e1042b7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:55,398 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188695398"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188695398"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188695398"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188695398"}]},"ts":"1689188695398"} 2023-07-12 19:04:55,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,398 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,399 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,400 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f1 2023-07-12 19:04:55,400 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f1 2023-07-12 19:04:55,400 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 30230a0350610d4e317e202c64d538a0 columnFamilyName f1 2023-07-12 19:04:55,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=35, resume processing ppid=31 2023-07-12 19:04:55,401 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, ppid=31, state=SUCCESS; OpenRegionProcedure b8c2da22e126648557cf9d684e1042b7, server=jenkins-hbase4.apache.org,41561,1689188688006 in 174 msec 2023-07-12 19:04:55,401 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(310): Store=30230a0350610d4e317e202c64d538a0/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,401 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,402 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f2 2023-07-12 19:04:55,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=b8c2da22e126648557cf9d684e1042b7, ASSIGN in 346 msec 2023-07-12 19:04:55,402 DEBUG [StoreOpener-30230a0350610d4e317e202c64d538a0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/f2 2023-07-12 19:04:55,402 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 30230a0350610d4e317e202c64d538a0 columnFamilyName f2 2023-07-12 19:04:55,403 INFO [StoreOpener-30230a0350610d4e317e202c64d538a0-1] regionserver.HStore(310): Store=30230a0350610d4e317e202c64d538a0/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:55,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 30230a0350610d4e317e202c64d538a0; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64393196, jitterRate=-0.04046660661697388}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:55,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 30230a0350610d4e317e202c64d538a0: 2023-07-12 19:04:55,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0., pid=33, masterSystemTime=1689188695367 2023-07-12 19:04:55,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:55,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5a9c075248bfea33c3c629a9f775522e, NAME => 'testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.', STARTKEY => '', ENDKEY => 'a'} 2023-07-12 19:04:55,411 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=30230a0350610d4e317e202c64d538a0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testEnableTableWithFailoverAtStep3 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,411 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1689188695410"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188695410"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188695410"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188695410"}]},"ts":"1689188695410"} 2023-07-12 19:04:55,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-12 19:04:55,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,412 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f1 of region 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,413 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f1 2023-07-12 19:04:55,413 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f1 2023-07-12 19:04:55,413 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a9c075248bfea33c3c629a9f775522e columnFamilyName f1 2023-07-12 19:04:55,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=29 2023-07-12 19:04:55,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=29, state=SUCCESS; OpenRegionProcedure 30230a0350610d4e317e202c64d538a0, server=jenkins-hbase4.apache.org,41561,1689188688006 in 194 msec 2023-07-12 19:04:55,414 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(310): Store=5a9c075248bfea33c3c629a9f775522e/f1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,414 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f2 of region 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,414 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=30230a0350610d4e317e202c64d538a0, ASSIGN in 358 msec 2023-07-12 19:04:55,414 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f2 2023-07-12 19:04:55,415 DEBUG [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/f2 2023-07-12 19:04:55,415 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5a9c075248bfea33c3c629a9f775522e columnFamilyName f2 2023-07-12 19:04:55,415 INFO [StoreOpener-5a9c075248bfea33c3c629a9f775522e-1] regionserver.HStore(310): Store=5a9c075248bfea33c3c629a9f775522e/f2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-12 19:04:55,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testEnableTableWithFailoverAtStep3 descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-07-12 19:04:55,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5a9c075248bfea33c3c629a9f775522e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60201697, jitterRate=-0.10292480885982513}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-07-12 19:04:55,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5a9c075248bfea33c3c629a9f775522e: 2023-07-12 19:04:55,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e., pid=34, masterSystemTime=1689188695367 2023-07-12 19:04:55,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:55,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:55,422 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=5a9c075248bfea33c3c629a9f775522e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,422 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1689188695422"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689188695422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689188695422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689188695422"}]},"ts":"1689188695422"} 2023-07-12 19:04:55,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=28 2023-07-12 19:04:55,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=28, state=SUCCESS; OpenRegionProcedure 5a9c075248bfea33c3c629a9f775522e, server=jenkins-hbase4.apache.org,41561,1689188688006 in 201 msec 2023-07-12 19:04:55,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-12 19:04:55,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; TransitRegionStateProcedure table=testEnableTableWithFailoverAtStep3, region=5a9c075248bfea33c3c629a9f775522e, ASSIGN in 369 msec 2023-07-12 19:04:55,426 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testEnableTableWithFailoverAtStep3","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689188695426"}]},"ts":"1689188695426"} 2023-07-12 19:04:55,427 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testEnableTableWithFailoverAtStep3, state=ENABLED in hbase:meta 2023-07-12 19:04:55,431 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testEnableTableWithFailoverAtStep3' was successfully enabled. 2023-07-12 19:04:55,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; EnableTableProcedure table=testEnableTableWithFailoverAtStep3 in 2.4170 sec 2023-07-12 19:04:55,560 INFO [Listener at localhost/44411] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-12 19:04:55,561 INFO [Listener at localhost/44411] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-12 19:04:55,561 DEBUG [Listener at localhost/44411] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a84d042 to 127.0.0.1:63125 2023-07-12 19:04:55,561 DEBUG [Listener at localhost/44411] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:55,561 DEBUG [Listener at localhost/44411] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-12 19:04:55,561 DEBUG [Listener at localhost/44411] util.JVMClusterUtil(257): Found active master hash=1139937625, stopped=false 2023-07-12 19:04:55,561 INFO [Listener at localhost/44411] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:55,563 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:55,563 INFO [Listener at localhost/44411] procedure2.ProcedureExecutor(629): Stopping 2023-07-12 19:04:55,563 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-12 19:04:55,563 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:55,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:55,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-12 19:04:55,563 DEBUG [Listener at localhost/44411] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d6b724 to 127.0.0.1:63125 2023-07-12 19:04:55,564 DEBUG [Listener at localhost/44411] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:55,564 INFO [Listener at localhost/44411] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41561,1689188688006' ***** 2023-07-12 19:04:55,564 INFO [Listener at localhost/44411] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-12 19:04:55,564 INFO [RS:0;jenkins-hbase4:41561] regionserver.HeapMemoryManager(220): Stopping 2023-07-12 19:04:55,564 INFO [RS:0;jenkins-hbase4:41561] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-12 19:04:55,564 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-12 19:04:55,564 INFO [RS:0;jenkins-hbase4:41561] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for 30230a0350610d4e317e202c64d538a0 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for 7e477b68840adb0e5568a48b431b3f77 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for 5a9c075248bfea33c3c629a9f775522e 2023-07-12 19:04:55,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 30230a0350610d4e317e202c64d538a0, disabling compactions & flushes 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,565 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,565 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:55,565 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,566 DEBUG [RS:0;jenkins-hbase4:41561] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cea50c2 to 127.0.0.1:63125 2023-07-12 19:04:55,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. after waiting 0 ms 2023-07-12 19:04:55,566 DEBUG [RS:0;jenkins-hbase4:41561] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:55,566 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,566 INFO [RS:0;jenkins-hbase4:41561] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-12 19:04:55,566 INFO [RS:0;jenkins-hbase4:41561] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-12 19:04:55,566 INFO [RS:0;jenkins-hbase4:41561] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-12 19:04:55,566 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-12 19:04:55,566 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1474): Waiting on 6 regions to close 2023-07-12 19:04:55,566 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1478): Online Regions={30230a0350610d4e317e202c64d538a0=testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0., 1588230740=hbase:meta,,1.1588230740, fdca952efa3dd0e15362758cf1ed720d=testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d., 7e477b68840adb0e5568a48b431b3f77=hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77., 5a9c075248bfea33c3c629a9f775522e=testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e., b8c2da22e126648557cf9d684e1042b7=testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7.} 2023-07-12 19:04:55,566 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1504): Waiting on 1588230740, 30230a0350610d4e317e202c64d538a0, 5a9c075248bfea33c3c629a9f775522e, 7e477b68840adb0e5568a48b431b3f77, b8c2da22e126648557cf9d684e1042b7, fdca952efa3dd0e15362758cf1ed720d 2023-07-12 19:04:55,567 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-12 19:04:55,567 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-12 19:04:55,567 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-12 19:04:55,567 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-12 19:04:55,567 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-12 19:04:55,567 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=15.07 KB heapSize=24.05 KB 2023-07-12 19:04:55,570 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/30230a0350610d4e317e202c64d538a0/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 19:04:55,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 30230a0350610d4e317e202c64d538a0: 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,a,1689188691450.30230a0350610d4e317e202c64d538a0. 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fdca952efa3dd0e15362758cf1ed720d, disabling compactions & flushes 2023-07-12 19:04:55,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. after waiting 0 ms 2023-07-12 19:04:55,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,574 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/fdca952efa3dd0e15362758cf1ed720d/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 19:04:55,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fdca952efa3dd0e15362758cf1ed720d: 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,b,1689188691450.fdca952efa3dd0e15362758cf1ed720d. 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7e477b68840adb0e5568a48b431b3f77, disabling compactions & flushes 2023-07-12 19:04:55,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. after waiting 0 ms 2023-07-12 19:04:55,575 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:55,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7e477b68840adb0e5568a48b431b3f77 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-12 19:04:55,579 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:55,635 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-07-12 19:04:55,767 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1504): Waiting on 1588230740, 5a9c075248bfea33c3c629a9f775522e, 7e477b68840adb0e5568a48b431b3f77, b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,967 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1504): Waiting on 1588230740, 5a9c075248bfea33c3c629a9f775522e, 7e477b68840adb0e5568a48b431b3f77, b8c2da22e126648557cf9d684e1042b7 2023-07-12 19:04:55,976 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=14.53 KB at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/.tmp/info/703ad5041ae24a82acb80d276d1ba1fb 2023-07-12 19:04:55,982 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/.tmp/info/ac196d6039564d1e8ee9874f325218fd 2023-07-12 19:04:55,987 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:55,987 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-12 19:04:55,988 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/.tmp/info/ac196d6039564d1e8ee9874f325218fd as hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/info/ac196d6039564d1e8ee9874f325218fd 2023-07-12 19:04:55,992 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/info/ac196d6039564d1e8ee9874f325218fd, entries=2, sequenceid=6, filesize=4.8 K 2023-07-12 19:04:55,993 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 7e477b68840adb0e5568a48b431b3f77 in 418ms, sequenceid=6, compaction requested=false 2023-07-12 19:04:55,997 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/namespace/7e477b68840adb0e5568a48b431b3f77/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-07-12 19:04:55,998 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:55,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7e477b68840adb0e5568a48b431b3f77: 2023-07-12 19:04:55,998 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689188690175.7e477b68840adb0e5568a48b431b3f77. 2023-07-12 19:04:55,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5a9c075248bfea33c3c629a9f775522e, disabling compactions & flushes 2023-07-12 19:04:55,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:55,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:55,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. after waiting 0 ms 2023-07-12 19:04:55,999 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:56,001 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/5a9c075248bfea33c3c629a9f775522e/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 19:04:56,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5a9c075248bfea33c3c629a9f775522e: 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,,1689188691450.5a9c075248bfea33c3c629a9f775522e. 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b8c2da22e126648557cf9d684e1042b7, disabling compactions & flushes 2023-07-12 19:04:56,002 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. after waiting 0 ms 2023-07-12 19:04:56,002 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:56,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/default/testEnableTableWithFailoverAtStep3/b8c2da22e126648557cf9d684e1042b7/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-07-12 19:04:56,006 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:56,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b8c2da22e126648557cf9d684e1042b7: 2023-07-12 19:04:56,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed testEnableTableWithFailoverAtStep3,c,1689188691450.b8c2da22e126648557cf9d684e1042b7. 2023-07-12 19:04:56,074 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-12 19:04:56,074 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-12 19:04:56,167 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:56,368 DEBUG [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1504): Waiting on 1588230740 2023-07-12 19:04:56,389 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=556 B at sequenceid=41 (bloomFilter=false), to=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/.tmp/table/70f09d9107464890b3ecb505fac60731 2023-07-12 19:04:56,394 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/.tmp/info/703ad5041ae24a82acb80d276d1ba1fb as hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/info/703ad5041ae24a82acb80d276d1ba1fb 2023-07-12 19:04:56,399 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/info/703ad5041ae24a82acb80d276d1ba1fb, entries=70, sequenceid=41, filesize=14.7 K 2023-07-12 19:04:56,400 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/.tmp/table/70f09d9107464890b3ecb505fac60731 as hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/table/70f09d9107464890b3ecb505fac60731 2023-07-12 19:04:56,405 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/table/70f09d9107464890b3ecb505fac60731, entries=5, sequenceid=41, filesize=4.9 K 2023-07-12 19:04:56,406 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~15.07 KB/15431, heapSize ~23.77 KB/24336, currentSize=0 B/0 for 1588230740 in 839ms, sequenceid=41, compaction requested=false 2023-07-12 19:04:56,413 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/data/hbase/meta/1588230740/recovered.edits/44.seqid, newMaxSeqId=44, maxSeqId=1 2023-07-12 19:04:56,414 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-12 19:04:56,414 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:56,414 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-12 19:04:56,415 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-12 19:04:56,568 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41561,1689188688006; all regions closed. 2023-07-12 19:04:56,571 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/WALs/jenkins-hbase4.apache.org,41561,1689188688006/jenkins-hbase4.apache.org%2C41561%2C1689188688006.meta.1689188690117.meta not finished, retry = 0 2023-07-12 19:04:56,674 DEBUG [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/oldWALs 2023-07-12 19:04:56,674 INFO [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41561%2C1689188688006.meta:.meta(num 1689188690117) 2023-07-12 19:04:56,678 DEBUG [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/oldWALs 2023-07-12 19:04:56,678 INFO [RS:0;jenkins-hbase4:41561] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41561%2C1689188688006:(num 1689188689201) 2023-07-12 19:04:56,678 DEBUG [RS:0;jenkins-hbase4:41561] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:56,678 INFO [RS:0;jenkins-hbase4:41561] regionserver.LeaseManager(133): Closed leases 2023-07-12 19:04:56,678 INFO [RS:0;jenkins-hbase4:41561] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-12 19:04:56,678 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:56,679 INFO [RS:0;jenkins-hbase4:41561] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41561 2023-07-12 19:04:56,681 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41561,1689188688006 2023-07-12 19:04:56,681 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:56,681 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-12 19:04:56,682 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41561,1689188688006] 2023-07-12 19:04:56,683 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41561,1689188688006; numProcessing=1 2023-07-12 19:04:56,685 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41561,1689188688006 already deleted, retry=false 2023-07-12 19:04:56,685 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,41561,1689188688006 expired; onlineServers=0 2023-07-12 19:04:56,685 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42753,1689188687935' ***** 2023-07-12 19:04:56,685 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-12 19:04:56,685 DEBUG [M:0;jenkins-hbase4:42753] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47cd9dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-12 19:04:56,685 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,42753,1689188687935 2023-07-12 19:04:56,685 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42753,1689188687935; all regions closed. 2023-07-12 19:04:56,685 DEBUG [M:0;jenkins-hbase4:42753] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-12 19:04:56,685 DEBUG [M:0;jenkins-hbase4:42753] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-12 19:04:56,685 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-12 19:04:56,685 DEBUG [M:0;jenkins-hbase4:42753] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-12 19:04:56,685 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188695024] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689188695024,5,FailOnTimeoutGroup] 2023-07-12 19:04:56,686 INFO [M:0;jenkins-hbase4:42753] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-12 19:04:56,685 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188695024] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689188695024,5,FailOnTimeoutGroup] 2023-07-12 19:04:56,686 INFO [M:0;jenkins-hbase4:42753] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-12 19:04:56,687 INFO [M:0;jenkins-hbase4:42753] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-12 19:04:56,687 DEBUG [M:0;jenkins-hbase4:42753] master.HMaster(1512): Stopping service threads 2023-07-12 19:04:56,687 INFO [M:0;jenkins-hbase4:42753] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-12 19:04:56,687 ERROR [M:0;jenkins-hbase4:42753] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-12 19:04:56,687 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-12 19:04:56,687 INFO [M:0;jenkins-hbase4:42753] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-12 19:04:56,687 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-12 19:04:56,687 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-12 19:04:56,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-12 19:04:56,688 DEBUG [M:0;jenkins-hbase4:42753] zookeeper.ZKUtil(398): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-12 19:04:56,688 WARN [M:0;jenkins-hbase4:42753] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-12 19:04:56,688 INFO [M:0;jenkins-hbase4:42753] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-12 19:04:56,688 INFO [M:0;jenkins-hbase4:42753] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-12 19:04:56,689 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-12 19:04:56,689 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:56,689 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:56,689 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-12 19:04:56,689 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:56,689 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.96 KB heapSize=30.73 KB 2023-07-12 19:04:56,782 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:56,783 INFO [RS:0;jenkins-hbase4:41561] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41561,1689188688006; zookeeper connection closed. 2023-07-12 19:04:56,783 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): regionserver:41561-0x1015aec61a40002, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:56,783 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@372b4431] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@372b4431 2023-07-12 19:04:56,784 INFO [Listener at localhost/44411] util.JVMClusterUtil(335): Shutdown of 2 master(s) and 1 regionserver(s) complete 2023-07-12 19:04:57,097 INFO [M:0;jenkins-hbase4:42753] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.96 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/506ac08502f84441a041be8baa6a9ad0 2023-07-12 19:04:57,103 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/506ac08502f84441a041be8baa6a9ad0 as hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/506ac08502f84441a041be8baa6a9ad0 2023-07-12 19:04:57,107 INFO [M:0;jenkins-hbase4:42753] regionserver.HStore(1080): Added hdfs://localhost:33505/user/jenkins/test-data/bfa7ab0f-97aa-2dc8-fe0a-6bd16ea2f6ed/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/506ac08502f84441a041be8baa6a9ad0, entries=9, sequenceid=254, filesize=5.4 K 2023-07-12 19:04:57,108 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegion(2948): Finished flush of dataSize ~24.96 KB/25554, heapSize ~30.71 KB/31448, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 419ms, sequenceid=254, compaction requested=false 2023-07-12 19:04:57,108 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-12 19:04:57,108 DEBUG [M:0;jenkins-hbase4:42753] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-12 19:04:57,111 INFO [M:0;jenkins-hbase4:42753] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-12 19:04:57,111 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-12 19:04:57,112 INFO [M:0;jenkins-hbase4:42753] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42753 2023-07-12 19:04:57,114 DEBUG [M:0;jenkins-hbase4:42753] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42753,1689188687935 already deleted, retry=false 2023-07-12 19:04:57,215 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:57,215 INFO [M:0;jenkins-hbase4:42753] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42753,1689188687935; zookeeper connection closed. 2023-07-12 19:04:57,215 DEBUG [Listener at localhost/44411-EventThread] zookeeper.ZKWatcher(600): master:42753-0x1015aec61a40000, quorum=127.0.0.1:63125, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-12 19:04:57,216 WARN [Listener at localhost/44411] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-12 19:04:57,220 INFO [Listener at localhost/44411] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:57,324 WARN [BP-784352885-172.31.14.131-1689188687103 heartbeating to localhost/127.0.0.1:33505] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-12 19:04:57,324 WARN [BP-784352885-172.31.14.131-1689188687103 heartbeating to localhost/127.0.0.1:33505] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-784352885-172.31.14.131-1689188687103 (Datanode Uuid e7646b78-88d6-4cbc-ba92-654db454bb77) service to localhost/127.0.0.1:33505 2023-07-12 19:04:57,325 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3/dfs/data/data1/current/BP-784352885-172.31.14.131-1689188687103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:57,325 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/c0490116-1a93-e3f5-3854-1697f7a8eeac/cluster_16bc1faf-6feb-1988-29fc-ffc0df4d42c3/dfs/data/data2/current/BP-784352885-172.31.14.131-1689188687103] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-12 19:04:57,335 INFO [Listener at localhost/44411] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-12 19:04:57,446 INFO [Listener at localhost/44411] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-12 19:04:57,454 INFO [Listener at localhost/44411] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-07-12 19:04:57,466 INFO [Listener at localhost/44411] hbase.ResourceChecker(175): after: master.procedure.TestMasterFailoverWithProcedures#testEnableTableWithFailover Thread=148 (was 124) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-16-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/44411 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33505 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33505 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: ForkJoinPool-2-worker-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-16-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: RS-EventLoopGroup-16-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:33505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (474849474) connection to localhost/127.0.0.1:33505 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:33505 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-15-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-17-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=502 (was 481) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=229 (was 209) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=8139 (was 8245)