2023-07-20 04:05:23,872 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-20 04:05:24,094 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c 2023-07-20 04:05:24,113 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-20 04:05:24,113 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7, deleteOnExit=true 2023-07-20 04:05:24,114 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-20 04:05:24,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/test.cache.data in system properties and HBase conf 2023-07-20 04:05:24,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/hadoop.tmp.dir in system properties and HBase conf 2023-07-20 04:05:24,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/hadoop.log.dir in system properties and HBase conf 2023-07-20 04:05:24,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-20 04:05:24,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-20 04:05:24,116 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-20 04:05:24,225 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-20 04:05:24,604 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-20 04:05:24,608 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-20 04:05:24,608 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-20 04:05:24,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-20 04:05:24,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 04:05:24,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-20 04:05:24,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-20 04:05:24,610 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 04:05:24,610 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 04:05:24,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-20 04:05:24,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/nfs.dump.dir in system properties and HBase conf 2023-07-20 04:05:24,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/java.io.tmpdir in system properties and HBase conf 2023-07-20 04:05:24,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 04:05:24,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-20 04:05:24,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-20 04:05:25,145 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 04:05:25,149 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 04:05:25,437 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-20 04:05:25,611 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-20 04:05:25,631 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 04:05:25,672 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 04:05:25,704 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/java.io.tmpdir/Jetty_localhost_42093_hdfs____xg43a/webapp 2023-07-20 04:05:25,874 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42093 2023-07-20 04:05:25,885 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 04:05:25,886 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 04:05:26,329 WARN [Listener at localhost/39169] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 04:05:26,430 WARN [Listener at localhost/39169] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 04:05:26,448 WARN [Listener at localhost/39169] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 04:05:26,455 INFO [Listener at localhost/39169] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 04:05:26,459 INFO [Listener at localhost/39169] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/java.io.tmpdir/Jetty_localhost_37251_datanode____5w5d1t/webapp 2023-07-20 04:05:26,555 INFO [Listener at localhost/39169] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37251 2023-07-20 04:05:26,863 WARN [Listener at localhost/35975] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 04:05:26,888 WARN [Listener at localhost/35975] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 04:05:26,894 WARN [Listener at localhost/35975] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 04:05:26,897 INFO [Listener at localhost/35975] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 04:05:26,912 INFO [Listener at localhost/35975] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/java.io.tmpdir/Jetty_localhost_36219_datanode____8ym872/webapp 2023-07-20 04:05:27,014 INFO [Listener at localhost/35975] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36219 2023-07-20 04:05:27,024 WARN [Listener at localhost/35413] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 04:05:27,341 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf26fadd1b42edcba: Processing first storage report for DS-e59b37fd-6abe-41bb-85ad-3200d342458c from datanode 2885965a-a266-45eb-b8d9-d40c7b55938f 2023-07-20 04:05:27,343 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf26fadd1b42edcba: from storage DS-e59b37fd-6abe-41bb-85ad-3200d342458c node DatanodeRegistration(127.0.0.1:44485, datanodeUuid=2885965a-a266-45eb-b8d9-d40c7b55938f, infoPort=35863, infoSecurePort=0, ipcPort=35975, storageInfo=lv=-57;cid=testClusterID;nsid=1934842691;c=1689825925221), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-20 04:05:27,343 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4e15c71a97022351: Processing first storage report for DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa from datanode ce15205f-9767-442d-8e21-8bb4260140d2 2023-07-20 04:05:27,344 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4e15c71a97022351: from storage DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa node DatanodeRegistration(127.0.0.1:32897, datanodeUuid=ce15205f-9767-442d-8e21-8bb4260140d2, infoPort=35529, infoSecurePort=0, ipcPort=35413, storageInfo=lv=-57;cid=testClusterID;nsid=1934842691;c=1689825925221), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-20 04:05:27,344 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf26fadd1b42edcba: Processing first storage report for DS-90a65468-5d54-4a5e-8942-0432e1a34ca5 from datanode 2885965a-a266-45eb-b8d9-d40c7b55938f 2023-07-20 04:05:27,344 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf26fadd1b42edcba: from storage DS-90a65468-5d54-4a5e-8942-0432e1a34ca5 node DatanodeRegistration(127.0.0.1:44485, datanodeUuid=2885965a-a266-45eb-b8d9-d40c7b55938f, infoPort=35863, infoSecurePort=0, ipcPort=35975, storageInfo=lv=-57;cid=testClusterID;nsid=1934842691;c=1689825925221), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 04:05:27,344 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x4e15c71a97022351: Processing first storage report for DS-5adab29b-64df-4cb3-bb51-fe223389db9a from datanode ce15205f-9767-442d-8e21-8bb4260140d2 2023-07-20 04:05:27,344 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x4e15c71a97022351: from storage DS-5adab29b-64df-4cb3-bb51-fe223389db9a node DatanodeRegistration(127.0.0.1:32897, datanodeUuid=ce15205f-9767-442d-8e21-8bb4260140d2, infoPort=35529, infoSecurePort=0, ipcPort=35413, storageInfo=lv=-57;cid=testClusterID;nsid=1934842691;c=1689825925221), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 04:05:27,429 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c 2023-07-20 04:05:27,497 INFO [Listener at localhost/35413] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/zookeeper_0, clientPort=60043, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-20 04:05:27,512 INFO [Listener at localhost/35413] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60043 2023-07-20 04:05:27,519 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:27,521 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:28,174 INFO [Listener at localhost/35413] util.FSUtils(471): Created version file at hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269 with version=8 2023-07-20 04:05:28,175 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/hbase-staging 2023-07-20 04:05:28,493 INFO [Listener at localhost/35413] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-20 04:05:28,982 INFO [Listener at localhost/35413] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 04:05:29,015 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,016 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,016 INFO [Listener at localhost/35413] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 04:05:29,017 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,017 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 04:05:29,163 INFO [Listener at localhost/35413] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 04:05:29,234 DEBUG [Listener at localhost/35413] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-20 04:05:29,329 INFO [Listener at localhost/35413] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34081 2023-07-20 04:05:29,339 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,341 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,366 INFO [Listener at localhost/35413] zookeeper.RecoverableZooKeeper(93): Process identifier=master:34081 connecting to ZooKeeper ensemble=127.0.0.1:60043 2023-07-20 04:05:29,412 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:340810x0, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 04:05:29,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:34081-0x10180e7e43c0000 connected 2023-07-20 04:05:29,438 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 04:05:29,439 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 04:05:29,442 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 04:05:29,450 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34081 2023-07-20 04:05:29,452 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34081 2023-07-20 04:05:29,454 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34081 2023-07-20 04:05:29,456 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34081 2023-07-20 04:05:29,457 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34081 2023-07-20 04:05:29,464 INFO [Listener at localhost/35413] master.HMaster(444): hbase.rootdir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269, hbase.cluster.distributed=false 2023-07-20 04:05:29,559 INFO [Listener at localhost/35413] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 04:05:29,559 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,559 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,560 INFO [Listener at localhost/35413] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 04:05:29,560 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,560 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 04:05:29,567 INFO [Listener at localhost/35413] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 04:05:29,571 INFO [Listener at localhost/35413] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34413 2023-07-20 04:05:29,574 INFO [Listener at localhost/35413] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 04:05:29,583 DEBUG [Listener at localhost/35413] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 04:05:29,584 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,586 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,587 INFO [Listener at localhost/35413] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34413 connecting to ZooKeeper ensemble=127.0.0.1:60043 2023-07-20 04:05:29,591 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:344130x0, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 04:05:29,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34413-0x10180e7e43c0001 connected 2023-07-20 04:05:29,592 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 04:05:29,593 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 04:05:29,594 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 04:05:29,594 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34413 2023-07-20 04:05:29,597 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34413 2023-07-20 04:05:29,598 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34413 2023-07-20 04:05:29,599 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34413 2023-07-20 04:05:29,600 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34413 2023-07-20 04:05:29,613 INFO [Listener at localhost/35413] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 04:05:29,613 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,613 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,614 INFO [Listener at localhost/35413] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 04:05:29,614 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:29,614 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 04:05:29,614 INFO [Listener at localhost/35413] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 04:05:29,616 INFO [Listener at localhost/35413] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41139 2023-07-20 04:05:29,616 INFO [Listener at localhost/35413] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 04:05:29,618 DEBUG [Listener at localhost/35413] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 04:05:29,620 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,622 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,625 INFO [Listener at localhost/35413] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41139 connecting to ZooKeeper ensemble=127.0.0.1:60043 2023-07-20 04:05:29,628 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:411390x0, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 04:05:29,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:41139-0x10180e7e43c0002 connected 2023-07-20 04:05:29,629 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 04:05:29,630 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 04:05:29,630 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 04:05:29,631 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41139 2023-07-20 04:05:29,631 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41139 2023-07-20 04:05:29,631 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41139 2023-07-20 04:05:29,632 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41139 2023-07-20 04:05:29,632 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41139 2023-07-20 04:05:29,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:29,642 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 04:05:29,644 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:29,663 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 04:05:29,663 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 04:05:29,663 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 04:05:29,663 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:29,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 04:05:29,666 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,34081,1689825928319 from backup master directory 2023-07-20 04:05:29,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 04:05:29,669 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:29,669 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 04:05:29,670 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 04:05:29,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:29,673 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-20 04:05:29,675 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-20 04:05:29,760 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/hbase.id with ID: feaa7c7f-b088-4c63-b042-e4af8d34dc27 2023-07-20 04:05:29,810 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:29,830 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:29,894 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6b328f1e to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:29,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ab64397, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:29,962 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 04:05:29,965 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-20 04:05:29,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-20 04:05:29,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-20 04:05:29,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:29,998 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:30,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:30,051 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store-tmp 2023-07-20 04:05:30,112 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:30,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 04:05:30,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:30,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:30,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 04:05:30,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:30,113 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:30,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 04:05:30,116 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/WALs/jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:30,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34081%2C1689825928319, suffix=, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/WALs/jenkins-hbase4.apache.org,34081,1689825928319, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/oldWALs, maxLogs=10 2023-07-20 04:05:30,207 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:30,207 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:30,218 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:30,293 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/WALs/jenkins-hbase4.apache.org,34081,1689825928319/jenkins-hbase4.apache.org%2C34081%2C1689825928319.1689825930154 2023-07-20 04:05:30,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK], DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK]] 2023-07-20 04:05:30,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:30,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:30,298 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,299 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,360 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,368 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-20 04:05:30,398 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-20 04:05:30,410 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:30,416 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,418 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,432 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 04:05:30,436 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 04:05:30,438 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71266093, jitterRate=0.0619475394487381}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:30,438 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 04:05:30,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-20 04:05:30,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-20 04:05:30,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-20 04:05:30,463 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-20 04:05:30,465 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-20 04:05:30,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 43 msec 2023-07-20 04:05:30,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-20 04:05:30,545 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-20 04:05:30,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-20 04:05:30,579 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-20 04:05:30,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-20 04:05:30,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-20 04:05:30,588 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-20 04:05:30,592 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-20 04:05:30,596 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:30,597 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-20 04:05:30,598 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-20 04:05:30,609 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-20 04:05:30,613 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 04:05:30,613 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 04:05:30,613 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 04:05:30,614 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:30,614 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,34081,1689825928319, sessionid=0x10180e7e43c0000, setting cluster-up flag (Was=false) 2023-07-20 04:05:30,633 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:30,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-20 04:05:30,640 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:30,646 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:30,651 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-20 04:05:30,652 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:30,655 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.hbase-snapshot/.tmp 2023-07-20 04:05:30,736 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(951): ClusterId : feaa7c7f-b088-4c63-b042-e4af8d34dc27 2023-07-20 04:05:30,736 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(951): ClusterId : feaa7c7f-b088-4c63-b042-e4af8d34dc27 2023-07-20 04:05:30,741 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 04:05:30,741 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 04:05:30,748 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 04:05:30,748 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 04:05:30,748 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 04:05:30,748 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 04:05:30,755 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 04:05:30,755 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 04:05:30,756 DEBUG [RS:1;jenkins-hbase4:41139] zookeeper.ReadOnlyZKClient(139): Connect 0x76974f5d to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:30,757 DEBUG [RS:0;jenkins-hbase4:34413] zookeeper.ReadOnlyZKClient(139): Connect 0x63d9fff3 to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:30,763 DEBUG [RS:1;jenkins-hbase4:41139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7850bfff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:30,763 DEBUG [RS:0;jenkins-hbase4:34413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d1e0df8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:30,764 DEBUG [RS:1;jenkins-hbase4:41139] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c6f242d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 04:05:30,764 DEBUG [RS:0;jenkins-hbase4:34413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69219483, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 04:05:30,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-20 04:05:30,782 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 04:05:30,783 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689825960785 2023-07-20 04:05:30,787 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34413 2023-07-20 04:05:30,788 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-20 04:05:30,790 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:41139 2023-07-20 04:05:30,791 INFO [RS:1;jenkins-hbase4:41139] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 04:05:30,792 INFO [RS:1;jenkins-hbase4:41139] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 04:05:30,791 INFO [RS:0;jenkins-hbase4:34413] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 04:05:30,792 INFO [RS:0;jenkins-hbase4:34413] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 04:05:30,792 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 04:05:30,792 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 04:05:30,792 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 04:05:30,793 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-20 04:05:30,794 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34081,1689825928319 with isa=jenkins-hbase4.apache.org/172.31.14.131:41139, startcode=1689825929612 2023-07-20 04:05:30,794 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34081,1689825928319 with isa=jenkins-hbase4.apache.org/172.31.14.131:34413, startcode=1689825929558 2023-07-20 04:05:30,798 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 04:05:30,799 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-20 04:05:30,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-20 04:05:30,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-20 04:05:30,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-20 04:05:30,807 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-20 04:05:30,808 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,809 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-20 04:05:30,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-20 04:05:30,811 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-20 04:05:30,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-20 04:05:30,814 DEBUG [RS:1;jenkins-hbase4:41139] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 04:05:30,814 DEBUG [RS:0;jenkins-hbase4:34413] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 04:05:30,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-20 04:05:30,818 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689825930816,5,FailOnTimeoutGroup] 2023-07-20 04:05:30,819 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689825930818,5,FailOnTimeoutGroup] 2023-07-20 04:05:30,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,819 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-20 04:05:30,820 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,842 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 04:05:30,843 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 04:05:30,844 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269 2023-07-20 04:05:30,867 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:30,871 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 04:05:30,874 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info 2023-07-20 04:05:30,875 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 04:05:30,876 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:30,877 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 04:05:30,879 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/rep_barrier 2023-07-20 04:05:30,880 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 04:05:30,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:30,881 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 04:05:30,882 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53723, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 04:05:30,882 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59207, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 04:05:30,884 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table 2023-07-20 04:05:30,885 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 04:05:30,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:30,887 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:30,888 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:30,893 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 04:05:30,894 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34081] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:30,895 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 04:05:30,899 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 04:05:30,901 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62774991, jitterRate=-0.06457974016666412}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 04:05:30,901 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 04:05:30,901 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 04:05:30,901 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 04:05:30,901 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 04:05:30,901 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 04:05:30,901 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 04:05:30,903 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 04:05:30,903 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 04:05:30,905 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34081] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:30,910 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 04:05:30,910 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-20 04:05:30,911 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269 2023-07-20 04:05:30,911 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269 2023-07-20 04:05:30,911 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39169 2023-07-20 04:05:30,911 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39169 2023-07-20 04:05:30,911 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 04:05:30,911 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 04:05:30,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 04:05:30,920 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:30,921 DEBUG [RS:1;jenkins-hbase4:41139] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:30,921 DEBUG [RS:0;jenkins-hbase4:34413] zookeeper.ZKUtil(162): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:30,921 WARN [RS:1;jenkins-hbase4:41139] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 04:05:30,922 WARN [RS:0;jenkins-hbase4:34413] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 04:05:30,922 INFO [RS:1;jenkins-hbase4:41139] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:30,923 INFO [RS:0;jenkins-hbase4:34413] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:30,923 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:30,923 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:30,923 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34413,1689825929558] 2023-07-20 04:05:30,924 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41139,1689825929612] 2023-07-20 04:05:30,939 DEBUG [RS:1;jenkins-hbase4:41139] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:30,939 DEBUG [RS:0;jenkins-hbase4:34413] zookeeper.ZKUtil(162): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:30,940 DEBUG [RS:1;jenkins-hbase4:41139] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:30,940 DEBUG [RS:0;jenkins-hbase4:34413] zookeeper.ZKUtil(162): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:30,941 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 04:05:30,948 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-20 04:05:30,954 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 04:05:30,954 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 04:05:30,964 INFO [RS:0;jenkins-hbase4:34413] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 04:05:30,964 INFO [RS:1;jenkins-hbase4:41139] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 04:05:30,985 INFO [RS:1;jenkins-hbase4:41139] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 04:05:30,985 INFO [RS:0;jenkins-hbase4:34413] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 04:05:30,989 INFO [RS:1;jenkins-hbase4:41139] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 04:05:30,989 INFO [RS:0;jenkins-hbase4:34413] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 04:05:30,990 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,990 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,991 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 04:05:30,991 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 04:05:30,998 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,998 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:30,998 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,998 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:30,999 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,000 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,000 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,000 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,000 DEBUG [RS:1;jenkins-hbase4:41139] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,000 DEBUG [RS:0;jenkins-hbase4:34413] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:31,002 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,002 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,002 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,002 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,002 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,002 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,018 INFO [RS:1;jenkins-hbase4:41139] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 04:05:31,018 INFO [RS:0;jenkins-hbase4:34413] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 04:05:31,021 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41139,1689825929612-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,021 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34413,1689825929558-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,037 INFO [RS:1;jenkins-hbase4:41139] regionserver.Replication(203): jenkins-hbase4.apache.org,41139,1689825929612 started 2023-07-20 04:05:31,037 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,41139,1689825929612, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41139, sessionid=0x10180e7e43c0002 2023-07-20 04:05:31,038 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 04:05:31,038 DEBUG [RS:1;jenkins-hbase4:41139] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:31,038 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41139,1689825929612' 2023-07-20 04:05:31,038 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 04:05:31,039 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 04:05:31,039 INFO [RS:0;jenkins-hbase4:34413] regionserver.Replication(203): jenkins-hbase4.apache.org,34413,1689825929558 started 2023-07-20 04:05:31,039 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 04:05:31,039 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,34413,1689825929558, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34413, sessionid=0x10180e7e43c0001 2023-07-20 04:05:31,040 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 04:05:31,040 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 04:05:31,040 DEBUG [RS:1;jenkins-hbase4:41139] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:31,040 DEBUG [RS:0;jenkins-hbase4:34413] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:31,040 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41139,1689825929612' 2023-07-20 04:05:31,041 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 04:05:31,040 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34413,1689825929558' 2023-07-20 04:05:31,041 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 04:05:31,041 DEBUG [RS:1;jenkins-hbase4:41139] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 04:05:31,041 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 04:05:31,042 DEBUG [RS:1;jenkins-hbase4:41139] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 04:05:31,042 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 04:05:31,042 INFO [RS:1;jenkins-hbase4:41139] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 04:05:31,042 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 04:05:31,042 INFO [RS:1;jenkins-hbase4:41139] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 04:05:31,042 DEBUG [RS:0;jenkins-hbase4:34413] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:31,043 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34413,1689825929558' 2023-07-20 04:05:31,043 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 04:05:31,044 DEBUG [RS:0;jenkins-hbase4:34413] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 04:05:31,044 DEBUG [RS:0;jenkins-hbase4:34413] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 04:05:31,044 INFO [RS:0;jenkins-hbase4:34413] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 04:05:31,045 INFO [RS:0;jenkins-hbase4:34413] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 04:05:31,100 DEBUG [jenkins-hbase4:34081] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-20 04:05:31,103 DEBUG [jenkins-hbase4:34081] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 04:05:31,109 DEBUG [jenkins-hbase4:34081] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 04:05:31,110 DEBUG [jenkins-hbase4:34081] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 04:05:31,110 DEBUG [jenkins-hbase4:34081] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 04:05:31,112 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41139,1689825929612, state=OPENING 2023-07-20 04:05:31,119 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-20 04:05:31,121 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:31,122 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 04:05:31,126 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41139,1689825929612}] 2023-07-20 04:05:31,157 INFO [RS:1;jenkins-hbase4:41139] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41139%2C1689825929612, suffix=, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs, maxLogs=32 2023-07-20 04:05:31,157 INFO [RS:0;jenkins-hbase4:34413] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34413%2C1689825929558, suffix=, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs, maxLogs=32 2023-07-20 04:05:31,183 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:31,184 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:31,190 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:31,190 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:31,196 INFO [RS:0;jenkins-hbase4:34413] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 2023-07-20 04:05:31,196 INFO [RS:1;jenkins-hbase4:41139] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 2023-07-20 04:05:31,196 DEBUG [RS:0;jenkins-hbase4:34413] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK], DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK]] 2023-07-20 04:05:31,197 DEBUG [RS:1;jenkins-hbase4:41139] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK], DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK]] 2023-07-20 04:05:31,308 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:31,311 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 04:05:31,314 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 04:05:31,327 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 04:05:31,327 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:31,331 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta, suffix=.meta, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs, maxLogs=32 2023-07-20 04:05:31,350 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:31,352 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:31,361 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta 2023-07-20 04:05:31,361 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK], DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK]] 2023-07-20 04:05:31,362 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:31,363 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 04:05:31,379 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 04:05:31,414 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 04:05:31,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 04:05:31,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:31,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 04:05:31,421 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 04:05:31,425 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 04:05:31,426 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info 2023-07-20 04:05:31,427 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info 2023-07-20 04:05:31,427 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 04:05:31,428 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:31,428 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 04:05:31,430 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/rep_barrier 2023-07-20 04:05:31,430 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/rep_barrier 2023-07-20 04:05:31,430 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 04:05:31,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:31,431 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 04:05:31,433 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table 2023-07-20 04:05:31,433 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table 2023-07-20 04:05:31,433 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 04:05:31,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:31,436 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:31,439 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:31,442 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 04:05:31,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 04:05:31,446 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65674302, jitterRate=-0.02137663960456848}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 04:05:31,446 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 04:05:31,455 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689825931301 2023-07-20 04:05:31,473 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 04:05:31,473 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 04:05:31,474 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,41139,1689825929612, state=OPEN 2023-07-20 04:05:31,477 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 04:05:31,477 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 04:05:31,480 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-20 04:05:31,481 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,41139,1689825929612 in 351 msec 2023-07-20 04:05:31,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-20 04:05:31,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 563 msec 2023-07-20 04:05:31,490 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 791 msec 2023-07-20 04:05:31,490 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689825931490, completionTime=-1 2023-07-20 04:05:31,490 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-20 04:05:31,491 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-20 04:05:31,560 DEBUG [hconnection-0x29b74c1e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:31,563 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42484, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:31,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-20 04:05:31,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689825991580 2023-07-20 04:05:31,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689826051580 2023-07-20 04:05:31,580 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 89 msec 2023-07-20 04:05:31,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34081,1689825928319-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34081,1689825928319-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34081,1689825928319-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:34081, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:31,612 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-20 04:05:31,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-20 04:05:31,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 04:05:31,633 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-20 04:05:31,635 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 04:05:31,638 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 04:05:31,660 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:31,663 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 empty. 2023-07-20 04:05:31,664 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:31,664 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-20 04:05:31,699 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-20 04:05:31,701 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 74636bf7e7eed2ee6e33191203c66a68, NAME => 'hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 74636bf7e7eed2ee6e33191203c66a68, disabling compactions & flushes 2023-07-20 04:05:31,719 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. after waiting 0 ms 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:31,719 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:31,719 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:31,724 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 04:05:31,742 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825931727"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689825931727"}]},"ts":"1689825931727"} 2023-07-20 04:05:31,772 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 04:05:31,774 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 04:05:31,780 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689825931774"}]},"ts":"1689825931774"} 2023-07-20 04:05:31,785 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-20 04:05:31,790 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 04:05:31,792 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 04:05:31,792 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 04:05:31,792 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 04:05:31,794 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN}] 2023-07-20 04:05:31,797 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN 2023-07-20 04:05:31,800 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34413,1689825929558; forceNewPlan=false, retain=false 2023-07-20 04:05:31,954 INFO [jenkins-hbase4:34081] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 04:05:31,955 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:31,955 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825931954"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825931954"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825931954"}]},"ts":"1689825931954"} 2023-07-20 04:05:31,960 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,34413,1689825929558}] 2023-07-20 04:05:32,114 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:32,115 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 04:05:32,118 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 04:05:32,125 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:32,126 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 74636bf7e7eed2ee6e33191203c66a68, NAME => 'hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:32,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:32,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,127 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,130 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,132 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:32,133 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:32,133 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74636bf7e7eed2ee6e33191203c66a68 columnFamilyName info 2023-07-20 04:05:32,134 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(310): Store=74636bf7e7eed2ee6e33191203c66a68/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:32,137 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,138 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,144 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:32,148 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 04:05:32,149 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 74636bf7e7eed2ee6e33191203c66a68; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69827495, jitterRate=0.04051075875759125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:32,149 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:32,152 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68., pid=6, masterSystemTime=1689825932114 2023-07-20 04:05:32,159 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:32,159 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:32,161 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:32,161 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825932160"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825932160"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825932160"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825932160"}]},"ts":"1689825932160"} 2023-07-20 04:05:32,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-20 04:05:32,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,34413,1689825929558 in 208 msec 2023-07-20 04:05:32,176 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-20 04:05:32,177 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN in 379 msec 2023-07-20 04:05:32,179 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 04:05:32,179 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689825932179"}]},"ts":"1689825932179"} 2023-07-20 04:05:32,183 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-20 04:05:32,187 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 04:05:32,190 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 564 msec 2023-07-20 04:05:32,236 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-20 04:05:32,238 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-20 04:05:32,238 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:32,267 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:32,270 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:32,284 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-20 04:05:32,304 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 04:05:32,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-07-20 04:05:32,317 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-20 04:05:32,329 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 04:05:32,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-20 04:05:32,344 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-20 04:05:32,347 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-20 04:05:32,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.678sec 2023-07-20 04:05:32,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-20 04:05:32,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-20 04:05:32,353 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-20 04:05:32,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34081,1689825928319-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-20 04:05:32,355 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34081,1689825928319-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-20 04:05:32,366 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-20 04:05:32,442 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(139): Connect 0x0d639341 to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:32,448 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53e20c54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:32,468 DEBUG [hconnection-0x76bfe176-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:32,485 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42490, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:32,498 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:32,512 DEBUG [Listener at localhost/35413] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-20 04:05:32,516 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36818, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-20 04:05:32,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 04:05:32,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-20 04:05:32,537 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 04:05:32,540 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 04:05:32,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-20 04:05:32,543 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,544 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/default/t/c3715463d679faec2126c0cd22263445 empty. 2023-07-20 04:05:32,546 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,546 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-20 04:05:32,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 04:05:32,571 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-20 04:05:32,573 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => c3715463d679faec2126c0cd22263445, NAME => 't,,1689825932528.c3715463d679faec2126c0cd22263445.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/.tmp 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689825932528.c3715463d679faec2126c0cd22263445.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing c3715463d679faec2126c0cd22263445, disabling compactions & flushes 2023-07-20 04:05:32,594 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. after waiting 0 ms 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,594 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,594 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:32,599 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 04:05:32,601 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825932601"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689825932601"}]},"ts":"1689825932601"} 2023-07-20 04:05:32,604 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 04:05:32,606 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 04:05:32,606 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689825932606"}]},"ts":"1689825932606"} 2023-07-20 04:05:32,609 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-20 04:05:32,613 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 04:05:32,615 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 04:05:32,615 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 04:05:32,615 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 04:05:32,615 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN}] 2023-07-20 04:05:32,618 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN 2023-07-20 04:05:32,620 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34413,1689825929558; forceNewPlan=false, retain=false 2023-07-20 04:05:32,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 04:05:32,770 INFO [jenkins-hbase4:34081] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 04:05:32,771 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:32,772 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825932771"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825932771"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825932771"}]},"ts":"1689825932771"} 2023-07-20 04:05:32,775 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,34413,1689825929558}] 2023-07-20 04:05:32,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 04:05:32,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c3715463d679faec2126c0cd22263445, NAME => 't,,1689825932528.c3715463d679faec2126c0cd22263445.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:32,936 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689825932528.c3715463d679faec2126c0cd22263445.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:32,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,939 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,942 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:32,943 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:32,943 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c3715463d679faec2126c0cd22263445 columnFamilyName f 2023-07-20 04:05:32,944 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(310): Store=c3715463d679faec2126c0cd22263445/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:32,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:32,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 04:05:32,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c3715463d679faec2126c0cd22263445; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72467611, jitterRate=0.07985155284404755}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:32,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:32,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689825932528.c3715463d679faec2126c0cd22263445., pid=11, masterSystemTime=1689825932928 2023-07-20 04:05:32,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:32,962 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:32,962 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825932962"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825932962"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825932962"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825932962"}]},"ts":"1689825932962"} 2023-07-20 04:05:32,969 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-20 04:05:32,969 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,34413,1689825929558 in 190 msec 2023-07-20 04:05:32,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-20 04:05:32,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN in 354 msec 2023-07-20 04:05:32,974 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 04:05:32,974 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689825932974"}]},"ts":"1689825932974"} 2023-07-20 04:05:32,977 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-20 04:05:32,980 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 04:05:32,982 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 448 msec 2023-07-20 04:05:33,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 04:05:33,163 INFO [Listener at localhost/35413] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-20 04:05:33,163 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-20 04:05:33,165 INFO [Listener at localhost/35413] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 04:05:33,171 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-20 04:05:33,171 INFO [Listener at localhost/35413] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 04:05:33,172 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-20 04:05:33,176 DEBUG [Listener at localhost/35413] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:33,179 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46162, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:33,232 INFO [Listener at localhost/35413] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=292, OpenFileDescriptor=587, MaxFileDescriptor=60000, SystemLoadAverage=138, ProcessCount=173, AvailableMemoryMB=6796 2023-07-20 04:05:33,234 INFO [Listener at localhost/35413] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-20 04:05:33,238 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 04:05:33,239 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,239 INFO [Listener at localhost/35413] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-20 04:05:33,239 INFO [Listener at localhost/35413] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-20 04:05:33,242 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689825932179/Put/vlen=2/seqid=0} 2023-07-20 04:05:33,242 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-20 04:05:33,242 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:regioninfo/1689825932160/Put/vlen=41/seqid=0, hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:seqnumDuringOpen/1689825932160/Put/vlen=8/seqid=0, hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:server/1689825932160/Put/vlen=31/seqid=0, hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:serverstartcode/1689825932160/Put/vlen=8/seqid=0, hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:sn/1689825931954/Put/vlen=45/seqid=0, hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68./info:state/1689825932160/Put/vlen=4/seqid=0} 2023-07-20 04:05:33,243 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 74636bf7e7eed2ee6e33191203c66a68, NAME => 'hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:33,243 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689825932974/Put/vlen=2/seqid=0} 2023-07-20 04:05:33,243 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-20 04:05:33,243 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689825932528.c3715463d679faec2126c0cd22263445./info:regioninfo/1689825932962/Put/vlen=35/seqid=0, t,,1689825932528.c3715463d679faec2126c0cd22263445./info:seqnumDuringOpen/1689825932962/Put/vlen=8/seqid=0, t,,1689825932528.c3715463d679faec2126c0cd22263445./info:server/1689825932962/Put/vlen=31/seqid=0, t,,1689825932528.c3715463d679faec2126c0cd22263445./info:serverstartcode/1689825932962/Put/vlen=8/seqid=0, t,,1689825932528.c3715463d679faec2126c0cd22263445./info:sn/1689825932771/Put/vlen=45/seqid=0, t,,1689825932528.c3715463d679faec2126c0cd22263445./info:state/1689825932962/Put/vlen=4/seqid=0} 2023-07-20 04:05:33,243 INFO [Listener at localhost/35413] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => c3715463d679faec2126c0cd22263445, NAME => 't,,1689825932528.c3715463d679faec2126c0cd22263445.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:33,244 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(139): Connect 0x2a43ef0c to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:33,251 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f7b538a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:33,251 INFO [Listener at localhost/35413] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-20 04:05:33,251 INFO [Listener at localhost/35413] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-20 04:05:33,254 DEBUG [hconnection-0x6f3baee2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:33,257 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:33,262 DEBUG [hconnection-0x6f3baee2-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:33,267 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:33,268 INFO [Listener at localhost/35413] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-20 04:05:33,272 ERROR [Listener at localhost/35413] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,34413,1689825929558: die! ***** 2023-07-20 04:05:33,272 ERROR [Listener at localhost/35413] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 04:05:33,275 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 04:05:33,278 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 04:05:33,278 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 04:05:33,279 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 04:05:33,279 INFO [Listener at localhost/35413] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 877658112, "init": 513802240, "max": 2051014656, "used": 389213888 }, "NonHeapMemoryUsage": { "committed": 113074176, "init": 2555904, "max": -1, "used": 110974056 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-20 04:05:33,283 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34081] master.MasterRpcServices(612): jenkins-hbase4.apache.org,34413,1689825929558 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,34413,1689825929558: die! ***** 2023-07-20 04:05:33,286 INFO [Listener at localhost/35413] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34413,1689825929558' ***** 2023-07-20 04:05:33,286 INFO [Listener at localhost/35413] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 04:05:33,287 INFO [RS:0;jenkins-hbase4:34413] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 04:05:33,287 INFO [RS:0;jenkins-hbase4:34413] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 04:05:33,287 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 04:05:33,287 INFO [RS:0;jenkins-hbase4:34413] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 04:05:33,287 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(3305): Received CLOSE for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:33,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -8825641684239908862 2023-07-20 04:05:33,288 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(3305): Received CLOSE for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:33,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 04:05:33,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c3715463d679faec2126c0cd22263445, disabling compactions & flushes 2023-07-20 04:05:33,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:46174 deadline: 1689825993287, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting 2023-07-20 04:05:33,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,288 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:33,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,289 DEBUG [RS:0;jenkins-hbase4:34413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63d9fff3 to 127.0.0.1:60043 2023-07-20 04:05:33,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. after waiting 0 ms 2023-07-20 04:05:33,289 DEBUG [RS:0;jenkins-hbase4:34413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:33,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,289 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1474): Waiting on 2 regions to close 2023-07-20 04:05:33,289 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1478): Online Regions={c3715463d679faec2126c0cd22263445=t,,1689825932528.c3715463d679faec2126c0cd22263445., 74636bf7e7eed2ee6e33191203c66a68=hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.} 2023-07-20 04:05:33,290 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:33,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:33,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 74636bf7e7eed2ee6e33191203c66a68, disabling compactions & flushes 2023-07-20 04:05:33,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. after waiting 0 ms 2023-07-20 04:05:33,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,292 DEBUG [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1504): Waiting on 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:33,295 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:33,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:33,308 DEBUG [Listener at localhost/35413] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 04:05:33,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 04:05:33,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:46174 deadline: 1689825993311, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting 2023-07-20 04:05:33,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 04:05:33,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34413] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:46174 deadline: 1689825993421, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,34413,1689825929558 aborting 2023-07-20 04:05:33,492 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34413,1689825929558; all regions closed. 2023-07-20 04:05:33,504 DEBUG [RS:0;jenkins-hbase4:34413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:33,504 INFO [RS:0;jenkins-hbase4:34413] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:33,505 INFO [RS:0;jenkins-hbase4:34413] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 04:05:33,505 INFO [RS:0;jenkins-hbase4:34413] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 04:05:33,505 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 04:05:33,505 INFO [RS:0;jenkins-hbase4:34413] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 04:05:33,505 INFO [RS:0;jenkins-hbase4:34413] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 04:05:33,506 INFO [RS:0;jenkins-hbase4:34413] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34413 2023-07-20 04:05:33,514 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:33,515 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:33,515 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:33,515 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:33,515 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:33,518 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34413,1689825929558] 2023-07-20 04:05:33,518 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34413,1689825929558; numProcessing=1 2023-07-20 04:05:33,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,519 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,34413,1689825929558 znode expired, triggering replicatorRemoved event 2023-07-20 04:05:33,521 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34413,1689825929558 already deleted, retry=false 2023-07-20 04:05:33,521 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,34413,1689825929558 on jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:33,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,529 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,34413,1689825929558, splitWal=true, meta=false 2023-07-20 04:05:33,529 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,34413,1689825929558 (carryingMeta=false) jenkins-hbase4.apache.org,34413,1689825929558/CRASHED/regionCount=2/lock=java.util.concurrent.locks.ReentrantReadWriteLock@50f3361c[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 04:05:33,532 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34413,1689825929558, splitWal=true, meta=false 2023-07-20 04:05:33,533 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,34413,1689825929558 had 2 regions 2023-07-20 04:05:33,536 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,34413,1689825929558, splitWal=true, meta=false, isMeta: false 2023-07-20 04:05:33,538 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting 2023-07-20 04:05:33,540 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,34413,1689825929558 WAL count=1, meta=false 2023-07-20 04:05:33,544 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160}] 2023-07-20 04:05:33,552 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,553 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, worker=jenkins-hbase4.apache.org,41139,1689825929612}] 2023-07-20 04:05:33,618 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:33,619 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:34413-0x10180e7e43c0001, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:33,619 INFO [RS:0;jenkins-hbase4:34413] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34413,1689825929558; zookeeper connection closed. 2023-07-20 04:05:33,619 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4e0a9280] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4e0a9280 2023-07-20 04:05:33,658 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:34413 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34413 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:33,659 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:34413 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:34413 2023-07-20 04:05:33,719 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41139] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-20 04:05:33,743 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, size=1.6 K (1665bytes) 2023-07-20 04:05:33,744 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 2023-07-20 04:05:33,748 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 after 4ms 2023-07-20 04:05:33,755 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 took 14ms 2023-07-20 04:05:33,773 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 74636bf7e7eed2ee6e33191203c66a68: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-20 04:05:33,777 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for c3715463d679faec2126c0cd22263445: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-20 04:05:33,778 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 so closing down 2023-07-20 04:05:33,778 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 04:05:33,781 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-20 04:05:33,786 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp 2023-07-20 04:05:33,786 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp 2023-07-20 04:05:33,788 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 04:05:33,810 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-20 04:05:33,811 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp (wrote 10 edits, skipped 0 edits in 1 ms) 2023-07-20 04:05:33,812 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000005 2023-07-20 04:05:33,813 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000013 2023-07-20 04:05:33,814 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 58 ms; skipped=2; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-20 04:05:33,814 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, journal: Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, size=1.6 K (1665bytes) at 1689825933743Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 so closing down at 1689825933778 (+35 ms)Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp at 1689825933786 (+8 ms)Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp at 1689825933787 (+1 ms)3 split writer threads finished at 1689825933788 (+1 ms)Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689825933810 (+22 ms)Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp (wrote 10 edits, skipped 0 edits in 1 ms) at 1689825933811 (+1 ms)Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000005 at 1689825933812 (+1 ms)Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000013 at 1689825933813 (+1 ms)Processed 14 edits across 2 Regions in 58 ms; skipped=2; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689825933814 (+1 ms) 2023-07-20 04:05:33,815 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 2023-07-20 04:05:33,818 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-20 04:05:33,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.HMaster(3641): Remote procedure done, pid=14 2023-07-20 04:05:33,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] wal.WALSplitUtil(141): Moved hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting/jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160 to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs 2023-07-20 04:05:33,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-20 04:05:33,831 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, worker=jenkins-hbase4.apache.org,41139,1689825929612 in 275 msec 2023-07-20 04:05:33,833 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,837 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-20 04:05:33,837 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C34413%2C1689825929558.1689825931160, worker=jenkins-hbase4.apache.org,41139,1689825929612 in 291 msec 2023-07-20 04:05:33,839 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting dir is empty, no logs to split. 2023-07-20 04:05:33,839 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,34413,1689825929558 WAL count=0, meta=false 2023-07-20 04:05:33,839 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,34413,1689825929558 WAL splitting is done? wals=0, meta=false 2023-07-20 04:05:33,842 INFO [PEWorker-5] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase4.apache.org,34413,1689825929558 failed, ignore...File hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,34413,1689825929558-splitting does not exist. 2023-07-20 04:05:33,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN}] 2023-07-20 04:05:33,845 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN 2023-07-20 04:05:33,845 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN 2023-07-20 04:05:33,846 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 04:05:33,846 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 04:05:33,966 DEBUG [hconnection-0x6f3baee2-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:34413 this server is in the failed servers list 2023-07-20 04:05:33,996 DEBUG [jenkins-hbase4:34081] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 04:05:33,997 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,997 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:33,997 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825933997"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825933997"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825933997"}]},"ts":"1689825933997"} 2023-07-20 04:05:33,997 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825933997"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825933997"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825933997"}]},"ts":"1689825933997"} 2023-07-20 04:05:34,000 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,41139,1689825929612}] 2023-07-20 04:05:34,002 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=16, state=RUNNABLE; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,41139,1689825929612}] 2023-07-20 04:05:34,002 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:34,159 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:34,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c3715463d679faec2126c0cd22263445, NAME => 't,,1689825932528.c3715463d679faec2126c0cd22263445.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:34,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689825932528.c3715463d679faec2126c0cd22263445.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:34,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,160 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,162 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,164 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:34,164 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:34,164 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:34,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 74636bf7e7eed2ee6e33191203c66a68, NAME => 'hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:34,165 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c3715463d679faec2126c0cd22263445 columnFamilyName f 2023-07-20 04:05:34,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:34,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,166 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(310): Store=c3715463d679faec2126c0cd22263445/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:34,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,168 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,171 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,172 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:34,172 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:34,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000013 2023-07-20 04:05:34,173 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74636bf7e7eed2ee6e33191203c66a68 columnFamilyName info 2023-07-20 04:05:34,174 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(310): Store=74636bf7e7eed2ee6e33191203c66a68/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:34,178 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000013 2023-07-20 04:05:34,183 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c3715463d679faec2126c0cd22263445 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-20 04:05:34,184 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000005 2023-07-20 04:05:34,188 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000005 2023-07-20 04:05:34,189 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 74636bf7e7eed2ee6e33191203c66a68 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-20 04:05:34,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/.tmp/f/dd9ba2561a15476dbf011a674d6f1d4e 2023-07-20 04:05:34,244 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/.tmp/info/384d23462b764941955e7d3b6d8f054d 2023-07-20 04:05:34,290 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/.tmp/info/384d23462b764941955e7d3b6d8f054d as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info/384d23462b764941955e7d3b6d8f054d 2023-07-20 04:05:34,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/.tmp/f/dd9ba2561a15476dbf011a674d6f1d4e as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f/dd9ba2561a15476dbf011a674d6f1d4e 2023-07-20 04:05:34,302 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info/384d23462b764941955e7d3b6d8f054d, entries=2, sequenceid=5, filesize=4.8 K 2023-07-20 04:05:34,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f/dd9ba2561a15476dbf011a674d6f1d4e, entries=10, sequenceid=13, filesize=4.8 K 2023-07-20 04:05:34,303 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 74636bf7e7eed2ee6e33191203c66a68 in 114ms, sequenceid=5, compaction requested=false; wal=null 2023-07-20 04:05:34,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for c3715463d679faec2126c0cd22263445 in 120ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 04:05:34,305 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-20 04:05:34,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-20 04:05:34,308 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/0000000000000000013 2023-07-20 04:05:34,308 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/0000000000000000005 2023-07-20 04:05:34,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:34,316 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:34,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 04:05:34,321 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-20 04:05:34,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c3715463d679faec2126c0cd22263445; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71355801, jitterRate=0.06328429281711578}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:34,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:34,323 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 74636bf7e7eed2ee6e33191203c66a68; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72696743, jitterRate=0.08326588571071625}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:34,323 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:34,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689825932528.c3715463d679faec2126c0cd22263445., pid=18, masterSystemTime=1689825934153 2023-07-20 04:05:34,325 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68., pid=17, masterSystemTime=1689825934153 2023-07-20 04:05:34,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:34,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:34,328 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:34,328 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:34,328 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:34,328 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825934328"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825934328"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825934328"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825934328"}]},"ts":"1689825934328"} 2023-07-20 04:05:34,329 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:34,329 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825934329"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825934329"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825934329"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825934329"}]},"ts":"1689825934329"} 2023-07-20 04:05:34,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=16 2023-07-20 04:05:34,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=16, state=SUCCESS; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,41139,1689825929612 in 328 msec 2023-07-20 04:05:34,335 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-07-20 04:05:34,336 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,41139,1689825929612 in 332 msec 2023-07-20 04:05:34,337 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN in 490 msec 2023-07-20 04:05:34,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-20 04:05:34,339 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,34413,1689825929558 after splitting done 2023-07-20 04:05:34,339 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN in 492 msec 2023-07-20 04:05:34,339 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,34413,1689825929558 from processing; numProcessing=0 2023-07-20 04:05:34,342 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,34413,1689825929558, splitWal=true, meta=false in 816 msec 2023-07-20 04:05:34,489 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a43ef0c to 127.0.0.1:60043 2023-07-20 04:05:34,489 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:34,490 INFO [Listener at localhost/35413] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-20 04:05:34,505 INFO [Listener at localhost/35413] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=270 (was 292), OpenFileDescriptor=575 (was 587), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=138 (was 138), ProcessCount=173 (was 173), AvailableMemoryMB=6665 (was 6796) 2023-07-20 04:05:34,520 INFO [Listener at localhost/35413] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=270, OpenFileDescriptor=575, MaxFileDescriptor=60000, SystemLoadAverage=138, ProcessCount=173, AvailableMemoryMB=6664 2023-07-20 04:05:34,521 INFO [Listener at localhost/35413] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:34413 2023-07-20 04:05:34,534 INFO [Listener at localhost/35413] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 04:05:34,534 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:34,534 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:34,534 INFO [Listener at localhost/35413] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 04:05:34,535 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 04:05:34,535 INFO [Listener at localhost/35413] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 04:05:34,535 INFO [Listener at localhost/35413] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 04:05:34,537 INFO [Listener at localhost/35413] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42483 2023-07-20 04:05:34,537 INFO [Listener at localhost/35413] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 04:05:34,543 DEBUG [Listener at localhost/35413] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 04:05:34,544 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:34,546 INFO [Listener at localhost/35413] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 04:05:34,548 INFO [Listener at localhost/35413] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42483 connecting to ZooKeeper ensemble=127.0.0.1:60043 2023-07-20 04:05:34,553 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:424830x0, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 04:05:34,554 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(162): regionserver:424830x0, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 04:05:34,555 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(162): regionserver:424830x0, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-20 04:05:34,556 DEBUG [Listener at localhost/35413] zookeeper.ZKUtil(164): regionserver:424830x0, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 04:05:34,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42483-0x10180e7e43c0008 connected 2023-07-20 04:05:34,562 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42483 2023-07-20 04:05:34,565 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42483 2023-07-20 04:05:34,566 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42483 2023-07-20 04:05:34,567 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42483 2023-07-20 04:05:34,568 DEBUG [Listener at localhost/35413] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42483 2023-07-20 04:05:34,570 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(951): ClusterId : feaa7c7f-b088-4c63-b042-e4af8d34dc27 2023-07-20 04:05:34,586 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 04:05:34,590 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 04:05:34,590 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 04:05:34,597 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 04:05:34,598 DEBUG [RS:2;jenkins-hbase4:42483] zookeeper.ReadOnlyZKClient(139): Connect 0x19f94b03 to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:34,605 DEBUG [RS:2;jenkins-hbase4:42483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@173eb64c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:34,605 DEBUG [RS:2;jenkins-hbase4:42483] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13012073, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 04:05:34,614 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:42483 2023-07-20 04:05:34,614 INFO [RS:2;jenkins-hbase4:42483] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 04:05:34,614 INFO [RS:2;jenkins-hbase4:42483] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 04:05:34,614 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 04:05:34,615 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,34081,1689825928319 with isa=jenkins-hbase4.apache.org/172.31.14.131:42483, startcode=1689825934533 2023-07-20 04:05:34,615 DEBUG [RS:2;jenkins-hbase4:42483] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 04:05:34,619 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42849, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 04:05:34,620 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=34081] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,621 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269 2023-07-20 04:05:34,621 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:39169 2023-07-20 04:05:34,621 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 04:05:34,623 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:34,623 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:34,623 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42483,1689825934533] 2023-07-20 04:05:34,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,624 DEBUG [RS:2;jenkins-hbase4:42483] zookeeper.ZKUtil(162): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:34,624 WARN [RS:2;jenkins-hbase4:42483] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 04:05:34,625 INFO [RS:2;jenkins-hbase4:42483] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:34,625 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1948): logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,630 DEBUG [RS:2;jenkins-hbase4:42483] zookeeper.ZKUtil(162): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,631 DEBUG [RS:2;jenkins-hbase4:42483] zookeeper.ZKUtil(162): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:34,632 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 04:05:34,633 INFO [RS:2;jenkins-hbase4:42483] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 04:05:34,639 INFO [RS:2;jenkins-hbase4:42483] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 04:05:34,640 INFO [RS:2;jenkins-hbase4:42483] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 04:05:34,640 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,642 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 04:05:34,644 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,644 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,644 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,644 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,645 DEBUG [RS:2;jenkins-hbase4:42483] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 04:05:34,647 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,647 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,648 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,664 INFO [RS:2;jenkins-hbase4:42483] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 04:05:34,664 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42483,1689825934533-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 04:05:34,675 INFO [RS:2;jenkins-hbase4:42483] regionserver.Replication(203): jenkins-hbase4.apache.org,42483,1689825934533 started 2023-07-20 04:05:34,676 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,42483,1689825934533, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42483, sessionid=0x10180e7e43c0008 2023-07-20 04:05:34,676 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:42483,5,FailOnTimeoutGroup] 2023-07-20 04:05:34,676 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 04:05:34,676 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:34,676 DEBUG [RS:2;jenkins-hbase4:42483] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,676 INFO [Listener at localhost/35413] client.TestScannerTimeout(177): START ************ test3686b 2023-07-20 04:05:34,676 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42483,1689825934533' 2023-07-20 04:05:34,676 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 04:05:34,677 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 04:05:34,678 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 04:05:34,678 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 04:05:34,678 DEBUG [RS:2;jenkins-hbase4:42483] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:34,678 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42483,1689825934533' 2023-07-20 04:05:34,678 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 04:05:34,680 DEBUG [RS:2;jenkins-hbase4:42483] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 04:05:34,680 DEBUG [RS:2;jenkins-hbase4:42483] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 04:05:34,680 INFO [RS:2;jenkins-hbase4:42483] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 04:05:34,680 INFO [RS:2;jenkins-hbase4:42483] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 04:05:34,680 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 04:05:34,681 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:34,681 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(139): Connect 0x60505a67 to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:34,688 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55d47cdc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:34,691 DEBUG [hconnection-0x10ba4d60-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:34,694 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42516, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:34,783 INFO [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42483%2C1689825934533, suffix=, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,42483,1689825934533, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs, maxLogs=32 2023-07-20 04:05:34,805 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:34,805 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:34,810 INFO [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,42483,1689825934533/jenkins-hbase4.apache.org%2C42483%2C1689825934533.1689825934784 2023-07-20 04:05:34,810 DEBUG [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK], DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK]] 2023-07-20 04:05:36,942 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-20 04:05:36,992 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-20 04:05:38,487 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 04:05:38,488 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-20 04:05:49,700 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -7980807864631427054 expired clientIPAndPort=172.31.14.131:42516, userName=jenkins, regionInfo=t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41139] regionserver.RSRpcServices(3094): Client tried to access missing scanner -7980807864631427054 2023-07-20 04:05:51,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=41139] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:42516 deadline: 1689826011701, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-7980807864631427054'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-20 04:05:51,703 DEBUG [Listener at localhost/35413] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-7980807864631427054'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-7980807864631427054'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-7980807864631427054'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 04:05:51,707 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x60505a67 to 127.0.0.1:60043 2023-07-20 04:05:51,707 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:51,707 INFO [Listener at localhost/35413] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-20 04:05:51,726 INFO [Listener at localhost/35413] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=294 (was 270) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_289853339_17 at /127.0.0.1:55966 [Receiving block BP-1949644524-172.31.14.131-1689825925221:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x19f94b03 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/457454565.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x19f94b03-SendThread(127.0.0.1:60043) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:39169 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:42483-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x60505a67-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269-prefix:jenkins-hbase4.apache.org,42483,1689825934533 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1949644524-172.31.14.131-1689825925221:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x60505a67-SendThread(127.0.0.1:60043) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: jenkins-hbase4:42483Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x60505a67 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/457454565.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42483 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1949644524-172.31.14.131-1689825925221:blk_1073741843_1019, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60043@0x19f94b03-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS:2;jenkins-hbase4:42483 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_289853339_17 at /127.0.0.1:50022 [Receiving block BP-1949644524-172.31.14.131-1689825925221:blk_1073741843_1019] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=572 (was 575), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=120 (was 138), ProcessCount=171 (was 173), AvailableMemoryMB=6477 (was 6664) 2023-07-20 04:05:51,742 INFO [Listener at localhost/35413] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=294, OpenFileDescriptor=572, MaxFileDescriptor=60000, SystemLoadAverage=120, ProcessCount=171, AvailableMemoryMB=6476 2023-07-20 04:05:51,743 INFO [Listener at localhost/35413] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:34413 2023-07-20 04:05:51,743 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,34413,1689825929558 2023-07-20 04:05:51,743 INFO [Listener at localhost/35413] client.TestScannerTimeout(104): START************ test2772 2023-07-20 04:05:51,747 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 04:05:51,747 DEBUG [Listener at localhost/35413] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,748 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(139): Connect 0x776125a4 to 127.0.0.1:60043 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 04:05:51,755 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56417cbe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 04:05:51,755 ERROR [Listener at localhost/35413] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,41139,1689825929612: die! ***** 2023-07-20 04:05:51,756 ERROR [Listener at localhost/35413] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 04:05:51,756 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 04:05:51,756 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 04:05:51,763 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 04:05:51,765 DEBUG [Listener at localhost/35413] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 04:05:51,789 INFO [Listener at localhost/35413] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 981467136, "init": 513802240, "max": 2051014656, "used": 268967968 }, "NonHeapMemoryUsage": { "committed": 122576896, "init": 2555904, "max": -1, "used": 120260504 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13025, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 63, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 63, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 63, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 63, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 678, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 63, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16704 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:60043", "tag.serverName": "jenkins-hbase4.apache.org,34413,1689825929558", "tag.clusterId": "feaa7c7f-b088-4c63-b042-e4af8d34dc27", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689825929558, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 1251, "localBytesRead": 1251, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 3314, "ByteBuffAllocatorPoolAllocationBytes": 399360, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 18, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 4, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 2, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 12, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 2, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 12, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 4, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 2, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.596131420570933, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.2499889058076004, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-20 04:05:51,790 WARN [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34081] master.MasterRpcServices(612): jenkins-hbase4.apache.org,41139,1689825929612 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,41139,1689825929612: die! ***** 2023-07-20 04:05:51,794 INFO [Listener at localhost/35413] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,41139,1689825929612' ***** 2023-07-20 04:05:51,799 INFO [Listener at localhost/35413] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 04:05:51,800 INFO [RS:1;jenkins-hbase4:41139] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 04:05:51,800 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 04:05:51,800 INFO [RS:1;jenkins-hbase4:41139] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 04:05:51,800 INFO [RS:1;jenkins-hbase4:41139] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 04:05:51,800 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(3305): Received CLOSE for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:51,800 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(3305): Received CLOSE for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:51,801 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:51,801 DEBUG [RS:1;jenkins-hbase4:41139] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76974f5d to 127.0.0.1:60043 2023-07-20 04:05:51,801 DEBUG [RS:1;jenkins-hbase4:41139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:51,801 INFO [RS:1;jenkins-hbase4:41139] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 04:05:51,801 INFO [RS:1;jenkins-hbase4:41139] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 04:05:51,801 INFO [RS:1;jenkins-hbase4:41139] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 04:05:51,801 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 04:05:51,802 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 04:05:51,802 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, c3715463d679faec2126c0cd22263445=t,,1689825932528.c3715463d679faec2126c0cd22263445., 74636bf7e7eed2ee6e33191203c66a68=hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.} 2023-07-20 04:05:51,802 DEBUG [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1504): Waiting on 1588230740, 74636bf7e7eed2ee6e33191203c66a68, c3715463d679faec2126c0cd22263445 2023-07-20 04:05:51,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c3715463d679faec2126c0cd22263445, disabling compactions & flushes 2023-07-20 04:05:51,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,805 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 04:05:51,805 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. after waiting 0 ms 2023-07-20 04:05:51,806 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 04:05:51,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,806 DEBUG [hconnection-0x7cfe3418-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:51,806 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 04:05:51,807 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 04:05:51,807 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 04:05:51,810 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 4186 in region hbase:meta,,1.1588230740 2023-07-20 04:05:51,811 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 04:05:51,813 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 04:05:51,813 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 04:05:51,814 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 04:05:51,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:51,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:51,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 74636bf7e7eed2ee6e33191203c66a68, disabling compactions & flushes 2023-07-20 04:05:51,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:51,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:51,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. after waiting 0 ms 2023-07-20 04:05:51,816 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:51,816 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42316, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:51,817 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41139] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41139,1689825929612 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 04:05:51,817 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41139] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:42316 deadline: 1689826011817, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41139,1689825929612 aborting 2023-07-20 04:05:51,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:51,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:51,825 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:51,926 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41139] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41139,1689825929612 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 04:05:51,926 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=41139] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:42316 deadline: 1689826011925, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,41139,1689825929612 aborting 2023-07-20 04:05:52,003 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,41139,1689825929612; all regions closed. 2023-07-20 04:05:52,014 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-20 04:05:52,014 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-20 04:05:52,017 DEBUG [RS:1;jenkins-hbase4:41139] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:52,017 INFO [RS:1;jenkins-hbase4:41139] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:52,017 INFO [RS:1;jenkins-hbase4:41139] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-20 04:05:52,018 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 04:05:52,019 INFO [RS:1;jenkins-hbase4:41139] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41139 2023-07-20 04:05:52,027 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:52,027 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:52,027 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 2023-07-20 04:05:52,027 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:52,027 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:52,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,029 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,41139,1689825929612] 2023-07-20 04:05:52,029 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41139,1689825929612 znode expired, triggering replicatorRemoved event 2023-07-20 04:05:52,029 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,41139,1689825929612; numProcessing=1 2023-07-20 04:05:52,031 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,41139,1689825929612 already deleted, retry=false 2023-07-20 04:05:52,031 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,41139,1689825929612 on jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:52,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,032 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,41139,1689825929612, splitWal=true, meta=true 2023-07-20 04:05:52,032 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=19 for jenkins-hbase4.apache.org,41139,1689825929612 (carryingMeta=true) jenkins-hbase4.apache.org,41139,1689825929612/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@32ff7633[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 04:05:52,034 INFO [PEWorker-1] procedure.ServerCrashProcedure(161): Start pid=19, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41139,1689825929612, splitWal=true, meta=true 2023-07-20 04:05:52,036 INFO [PEWorker-1] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41139,1689825929612, splitWal=true, meta=true, isMeta: true 2023-07-20 04:05:52,040 DEBUG [PEWorker-1] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting 2023-07-20 04:05:52,043 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,41139,1689825929612 WAL count=1, meta=true 2023-07-20 04:05:52,043 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta}] 2023-07-20 04:05:52,046 DEBUG [PEWorker-2] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, worker=jenkins-hbase4.apache.org,42483,1689825934533}] 2023-07-20 04:05:52,129 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41139 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41139 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:52,129 INFO [RS:1;jenkins-hbase4:41139] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,41139,1689825929612; zookeeper connection closed. 2023-07-20 04:05:52,129 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:52,129 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:41139-0x10180e7e43c0002, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:52,130 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41139 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41139 2023-07-20 04:05:52,130 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4775dffe] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4775dffe 2023-07-20 04:05:52,200 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,200 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 04:05:52,201 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 04:05:52,202 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=42483] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=21 2023-07-20 04:05:52,217 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, size=5.1 K (5185bytes) 2023-07-20 04:05:52,217 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta 2023-07-20 04:05:52,217 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta after 0ms 2023-07-20 04:05:52,223 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta took 5ms 2023-07-20 04:05:52,228 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-20 04:05:52,230 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta so closing down 2023-07-20 04:05:52,230 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 04:05:52,234 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp 2023-07-20 04:05:52,234 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 04:05:52,248 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) 2023-07-20 04:05:52,250 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 04:05:52,250 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false 2023-07-20 04:05:52,250 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, journal: Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, size=5.1 K (5185bytes) at 1689825952217Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta so closing down at 1689825952230 (+13 ms)Creating recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp at 1689825952234 (+4 ms)3 split writer threads finished at 1689825952234Closed recovered edits writer path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp (wrote 14 edits, skipped 0 edits in 0 ms) at 1689825952248 (+14 ms)Rename recovered edits hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta.temp to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 at 1689825952250 (+2 ms)Processed 15 edits across 1 Regions in 27 ms; skipped=1; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, size=5.1 K, length=5185, corrupted=false, cancelled=false at 1689825952250 2023-07-20 04:05:52,250 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta 2023-07-20 04:05:52,251 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=21 2023-07-20 04:05:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.HMaster(3641): Remote procedure done, pid=21 2023-07-20 04:05:52,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] wal.WALSplitUtil(141): Moved hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs 2023-07-20 04:05:52,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-20 04:05:52,258 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, worker=jenkins-hbase4.apache.org,42483,1689825934533 in 208 msec 2023-07-20 04:05:52,259 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,262 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-07-20 04:05:52,262 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.meta.1689825931333.meta, worker=jenkins-hbase4.apache.org,42483,1689825934533 in 216 msec 2023-07-20 04:05:52,264 INFO [PEWorker-1] master.SplitLogManager(171): hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting dir is empty, no logs to split. 2023-07-20 04:05:52,264 INFO [PEWorker-1] master.SplitWALManager(106): jenkins-hbase4.apache.org,41139,1689825929612 WAL count=0, meta=true 2023-07-20 04:05:52,264 DEBUG [PEWorker-1] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41139,1689825929612 WAL splitting is done? wals=0, meta=true 2023-07-20 04:05:52,265 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 04:05:52,266 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 04:05:52,267 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=22, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 04:05:52,418 DEBUG [jenkins-hbase4:34081] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 04:05:52,418 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42483,1689825934533, state=OPENING 2023-07-20 04:05:52,420 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 04:05:52,420 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42483,1689825934533}] 2023-07-20 04:05:52,420 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 04:05:52,432 WARN [ReadOnlyZKClient-127.0.0.1:60043@0x776125a4] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 04:05:52,432 DEBUG [hconnection-0x7cfe3418-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:52,435 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41218, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:52,435 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42483] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:41218 deadline: 1689826012435, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,485 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:52,578 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 04:05:52,578 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 04:05:52,580 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42483%2C1689825934533.meta, suffix=.meta, logDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,42483,1689825934533, archiveDir=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs, maxLogs=32 2023-07-20 04:05:52,598 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK] 2023-07-20 04:05:52,598 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK] 2023-07-20 04:05:52,600 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,42483,1689825934533/jenkins-hbase4.apache.org%2C42483%2C1689825934533.meta.1689825952581.meta 2023-07-20 04:05:52,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:32897,DS-7fcf601f-b5da-423a-8743-1f4bbc5f56aa,DISK], DatanodeInfoWithStorage[127.0.0.1:44485,DS-e59b37fd-6abe-41bb-85ad-3200d342458c,DISK]] 2023-07-20 04:05:52,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:52,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 04:05:52,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 04:05:52,602 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 04:05:52,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 04:05:52,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:52,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 04:05:52,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 04:05:52,605 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 04:05:52,606 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info 2023-07-20 04:05:52,606 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info 2023-07-20 04:05:52,607 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 04:05:52,609 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:52,609 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 04:05:52,610 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/rep_barrier 2023-07-20 04:05:52,611 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/rep_barrier 2023-07-20 04:05:52,611 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 04:05:52,612 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:52,612 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 04:05:52,613 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table 2023-07-20 04:05:52,613 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table 2023-07-20 04:05:52,614 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 04:05:52,614 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:52,615 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:52,617 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740 2023-07-20 04:05:52,618 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 04:05:52,626 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 40, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=17, path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 04:05:52,626 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=4.09 KB heapSize=7.79 KB 2023-07-20 04:05:52,647 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.93 KB at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/info/4686a6fd747c43dc92e48402456c7a76 2023-07-20 04:05:52,669 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=17 (bloomFilter=false), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/table/bdf7a7aa396e447b8902ade9072ca378 2023-07-20 04:05:52,677 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/info/4686a6fd747c43dc92e48402456c7a76 as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info/4686a6fd747c43dc92e48402456c7a76 2023-07-20 04:05:52,685 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info/4686a6fd747c43dc92e48402456c7a76, entries=36, sequenceid=17, filesize=8.6 K 2023-07-20 04:05:52,686 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/table/bdf7a7aa396e447b8902ade9072ca378 as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table/bdf7a7aa396e447b8902ade9072ca378 2023-07-20 04:05:52,693 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/table/bdf7a7aa396e447b8902ade9072ca378, entries=4, sequenceid=17, filesize=4.7 K 2023-07-20 04:05:52,693 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~4.09 KB/4186, heapSize ~7.51 KB/7688, currentSize=0 B/0 for 1588230740 in 67ms, sequenceid=17, compaction requested=false; wal=null 2023-07-20 04:05:52,695 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/0000000000000000017 2023-07-20 04:05:52,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 04:05:52,699 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 04:05:52,702 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2023-07-20 04:05:52,703 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=18; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74602309, jitterRate=0.11166103184223175}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 04:05:52,703 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 04:05:52,704 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=23, masterSystemTime=1689825952573 2023-07-20 04:05:52,706 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 04:05:52,707 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 04:05:52,707 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42483,1689825934533, state=OPEN 2023-07-20 04:05:52,710 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 04:05:52,710 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 04:05:52,713 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-20 04:05:52,713 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42483,1689825934533 in 290 msec 2023-07-20 04:05:52,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=19 2023-07-20 04:05:52,715 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,41139,1689825929612 had 3 regions 2023-07-20 04:05:52,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 448 msec 2023-07-20 04:05:52,716 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=19, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,41139,1689825929612, splitWal=true, meta=true, isMeta: false 2023-07-20 04:05:52,719 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,41139,1689825929612 WAL count=1, meta=false 2023-07-20 04:05:52,719 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=19, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160}] 2023-07-20 04:05:52,721 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,722 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, worker=jenkins-hbase4.apache.org,42483,1689825934533}] 2023-07-20 04:05:52,875 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=42483] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-20 04:05:52,888 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, size=721 (721bytes) 2023-07-20 04:05:52,888 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 2023-07-20 04:05:52,889 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 after 1ms 2023-07-20 04:05:52,892 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 took 4ms 2023-07-20 04:05:52,896 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for c3715463d679faec2126c0cd22263445: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-20 04:05:52,897 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 74636bf7e7eed2ee6e33191203c66a68: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-20 04:05:52,897 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 so closing down 2023-07-20 04:05:52,897 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 04:05:52,898 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 04:05:52,898 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, size=721, length=721, corrupted=false, cancelled=false 2023-07-20 04:05:52,898 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, journal: Splitting hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, size=721 (721bytes) at 1689825952888Finishing writing output for hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 so closing down at 1689825952897 (+9 ms)3 split writer threads finished at 1689825952898 (+1 ms)Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, size=721, length=721, corrupted=false, cancelled=false at 1689825952898 2023-07-20 04:05:52,898 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 2023-07-20 04:05:52,898 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-20 04:05:52,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] master.HMaster(3641): Remote procedure done, pid=25 2023-07-20 04:05:52,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34081] wal.WALSplitUtil(141): Moved hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting/jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160 to hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs 2023-07-20 04:05:52,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-20 04:05:52,907 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, worker=jenkins-hbase4.apache.org,42483,1689825934533 in 182 msec 2023-07-20 04:05:52,908 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:52,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=19 2023-07-20 04:05:52,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=19, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C41139%2C1689825929612.1689825931160, worker=jenkins-hbase4.apache.org,42483,1689825934533 in 190 msec 2023-07-20 04:05:52,914 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/WALs/jenkins-hbase4.apache.org,41139,1689825929612-splitting dir is empty, no logs to split. 2023-07-20 04:05:52,914 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,41139,1689825929612 WAL count=0, meta=false 2023-07-20 04:05:52,914 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,41139,1689825929612 WAL splitting is done? wals=0, meta=false 2023-07-20 04:05:52,916 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN}, {pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN}] 2023-07-20 04:05:52,917 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN 2023-07-20 04:05:52,917 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN 2023-07-20 04:05:52,918 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 04:05:52,918 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=27, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 04:05:52,952 DEBUG [hconnection-0x7cfe3418-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41139 this server is in the failed servers list 2023-07-20 04:05:53,062 DEBUG [hconnection-0x7cfe3418-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41139 this server is in the failed servers list 2023-07-20 04:05:53,068 DEBUG [jenkins-hbase4:34081] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 04:05:53,068 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,068 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,069 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825953068"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825953068"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825953068"}]},"ts":"1689825953068"} 2023-07-20 04:05:53,069 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825953068"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689825953068"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689825953068"}]},"ts":"1689825953068"} 2023-07-20 04:05:53,069 WARN [RS-EventLoopGroup-4-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:41139 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41139 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 04:05:53,071 DEBUG [RS-EventLoopGroup-4-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:41139 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:41139 2023-07-20 04:05:53,173 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 04:05:53,175 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 04:05:53,180 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=27, state=RUNNABLE; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,42483,1689825934533}] 2023-07-20 04:05:53,182 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=26, state=RUNNABLE; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,42483,1689825934533}] 2023-07-20 04:05:53,269 DEBUG [hconnection-0x7cfe3418-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:41139 this server is in the failed servers list 2023-07-20 04:05:53,338 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 74636bf7e7eed2ee6e33191203c66a68, NAME => 'hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:53,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:53,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,339 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,341 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,342 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:53,342 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info 2023-07-20 04:05:53,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c3715463d679faec2126c0cd22263445, NAME => 't,,1689825932528.c3715463d679faec2126c0cd22263445.', STARTKEY => '', ENDKEY => ''} 2023-07-20 04:05:53,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689825932528.c3715463d679faec2126c0cd22263445.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 04:05:53,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,343 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 74636bf7e7eed2ee6e33191203c66a68 columnFamilyName info 2023-07-20 04:05:53,343 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,346 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,348 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:53,348 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f 2023-07-20 04:05:53,350 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c3715463d679faec2126c0cd22263445 columnFamilyName f 2023-07-20 04:05:53,362 DEBUG [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(539): loaded hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/info/384d23462b764941955e7d3b6d8f054d 2023-07-20 04:05:53,363 INFO [StoreOpener-74636bf7e7eed2ee6e33191203c66a68-1] regionserver.HStore(310): Store=74636bf7e7eed2ee6e33191203c66a68/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:53,364 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,365 DEBUG [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(539): loaded hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/f/dd9ba2561a15476dbf011a674d6f1d4e 2023-07-20 04:05:53,366 INFO [StoreOpener-c3715463d679faec2126c0cd22263445-1] regionserver.HStore(310): Store=c3715463d679faec2126c0cd22263445/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 04:05:53,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,367 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,374 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,378 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-20 04:05:53,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-20 04:05:53,380 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 74636bf7e7eed2ee6e33191203c66a68; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59370910, jitterRate=-0.11530449986457825}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:53,380 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:53,382 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68., pid=29, masterSystemTime=1689825953333 2023-07-20 04:05:53,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c3715463d679faec2126c0cd22263445; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66543469, jitterRate=-0.008425042033195496}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 04:05:53,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:53,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689825932528.c3715463d679faec2126c0cd22263445., pid=28, masterSystemTime=1689825953333 2023-07-20 04:05:53,384 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,384 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,385 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=74636bf7e7eed2ee6e33191203c66a68, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,385 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689825953385"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825953385"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825953385"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825953385"}]},"ts":"1689825953385"} 2023-07-20 04:05:53,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,387 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=c3715463d679faec2126c0cd22263445, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,388 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689825932528.c3715463d679faec2126c0cd22263445.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689825953387"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689825953387"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689825953387"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689825953387"}]},"ts":"1689825953387"} 2023-07-20 04:05:53,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=26 2023-07-20 04:05:53,392 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=26, state=SUCCESS; OpenRegionProcedure 74636bf7e7eed2ee6e33191203c66a68, server=jenkins-hbase4.apache.org,42483,1689825934533 in 207 msec 2023-07-20 04:05:53,394 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=27 2023-07-20 04:05:53,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=74636bf7e7eed2ee6e33191203c66a68, ASSIGN in 477 msec 2023-07-20 04:05:53,394 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=27, state=SUCCESS; OpenRegionProcedure c3715463d679faec2126c0cd22263445, server=jenkins-hbase4.apache.org,42483,1689825934533 in 211 msec 2023-07-20 04:05:53,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=19 2023-07-20 04:05:53,397 INFO [PEWorker-3] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,41139,1689825929612 after splitting done 2023-07-20 04:05:53,397 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=t, region=c3715463d679faec2126c0cd22263445, ASSIGN in 479 msec 2023-07-20 04:05:53,397 DEBUG [PEWorker-3] master.DeadServer(114): Removed jenkins-hbase4.apache.org,41139,1689825929612 from processing; numProcessing=0 2023-07-20 04:05:53,400 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,41139,1689825929612, splitWal=true, meta=true in 1.3660 sec 2023-07-20 04:05:53,581 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x776125a4 to 127.0.0.1:60043 2023-07-20 04:05:53,581 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,581 INFO [Listener at localhost/35413] client.TestScannerTimeout(124): END ************ test2772 2023-07-20 04:05:53,594 INFO [Listener at localhost/35413] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=275 (was 294), OpenFileDescriptor=572 (was 572), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=120 (was 120), ProcessCount=171 (was 171), AvailableMemoryMB=6390 (was 6476) 2023-07-20 04:05:53,594 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-20 04:05:53,595 INFO [Listener at localhost/35413] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-20 04:05:53,595 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d639341 to 127.0.0.1:60043 2023-07-20 04:05:53,595 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,595 DEBUG [Listener at localhost/35413] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-20 04:05:53,595 DEBUG [Listener at localhost/35413] util.JVMClusterUtil(257): Found active master hash=298451691, stopped=false 2023-07-20 04:05:53,595 INFO [Listener at localhost/35413] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:53,605 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 04:05:53,605 INFO [Listener at localhost/35413] procedure2.ProcedureExecutor(629): Stopping 2023-07-20 04:05:53,605 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 04:05:53,605 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:53,605 DEBUG [Listener at localhost/35413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b328f1e to 127.0.0.1:60043 2023-07-20 04:05:53,605 DEBUG [Listener at localhost/35413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 04:05:53,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 04:05:53,605 INFO [Listener at localhost/35413] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,42483,1689825934533' ***** 2023-07-20 04:05:53,605 INFO [Listener at localhost/35413] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-20 04:05:53,606 INFO [RS:2;jenkins-hbase4:42483] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 04:05:53,606 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 04:05:53,606 INFO [RS:2;jenkins-hbase4:42483] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-20 04:05:53,606 INFO [RS:2;jenkins-hbase4:42483] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-20 04:05:53,606 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(3305): Received CLOSE for c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,606 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(3305): Received CLOSE for 74636bf7e7eed2ee6e33191203c66a68 2023-07-20 04:05:53,607 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,607 DEBUG [RS:2;jenkins-hbase4:42483] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19f94b03 to 127.0.0.1:60043 2023-07-20 04:05:53,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c3715463d679faec2126c0cd22263445, disabling compactions & flushes 2023-07-20 04:05:53,607 DEBUG [RS:2;jenkins-hbase4:42483] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,607 INFO [RS:2;jenkins-hbase4:42483] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 04:05:53,607 INFO [RS:2;jenkins-hbase4:42483] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 04:05:53,607 INFO [RS:2;jenkins-hbase4:42483] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 04:05:53,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,608 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 04:05:53,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689825932528.c3715463d679faec2126c0cd22263445. after waiting 0 ms 2023-07-20 04:05:53,608 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,608 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 04:05:53,608 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1478): Online Regions={1588230740=hbase:meta,,1.1588230740, c3715463d679faec2126c0cd22263445=t,,1689825932528.c3715463d679faec2126c0cd22263445., 74636bf7e7eed2ee6e33191203c66a68=hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68.} 2023-07-20 04:05:53,608 DEBUG [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1504): Waiting on 1588230740, 74636bf7e7eed2ee6e33191203c66a68, c3715463d679faec2126c0cd22263445 2023-07-20 04:05:53,608 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 04:05:53,608 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 04:05:53,608 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 04:05:53,608 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 04:05:53,609 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 04:05:53,610 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 04:05:53,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/default/t/c3715463d679faec2126c0cd22263445/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-20 04:05:53,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c3715463d679faec2126c0cd22263445: 2023-07-20 04:05:53,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689825932528.c3715463d679faec2126c0cd22263445. 2023-07-20 04:05:53,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 74636bf7e7eed2ee6e33191203c66a68, disabling compactions & flushes 2023-07-20 04:05:53,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. after waiting 0 ms 2023-07-20 04:05:53,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/namespace/74636bf7e7eed2ee6e33191203c66a68/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-20 04:05:53,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 74636bf7e7eed2ee6e33191203c66a68: 2023-07-20 04:05:53,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689825931621.74636bf7e7eed2ee6e33191203c66a68. 2023-07-20 04:05:53,640 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=24 (bloomFilter=false), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/info/409dfc7f77b54cba9419558f0714de70 2023-07-20 04:05:53,648 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-20 04:05:53,648 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-20 04:05:53,649 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/.tmp/info/409dfc7f77b54cba9419558f0714de70 as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info/409dfc7f77b54cba9419558f0714de70 2023-07-20 04:05:53,652 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:53,656 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/info/409dfc7f77b54cba9419558f0714de70, entries=16, sequenceid=24, filesize=6.4 K 2023-07-20 04:05:53,658 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 49ms, sequenceid=24, compaction requested=false 2023-07-20 04:05:53,665 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/data/hbase/meta/1588230740/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=17 2023-07-20 04:05:53,666 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 04:05:53,666 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 04:05:53,666 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 04:05:53,666 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 04:05:53,808 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,42483,1689825934533; all regions closed. 2023-07-20 04:05:53,815 DEBUG [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs 2023-07-20 04:05:53,815 INFO [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42483%2C1689825934533.meta:.meta(num 1689825952581) 2023-07-20 04:05:53,824 DEBUG [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/oldWALs 2023-07-20 04:05:53,824 INFO [RS:2;jenkins-hbase4:42483] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42483%2C1689825934533:(num 1689825934784) 2023-07-20 04:05:53,824 DEBUG [RS:2;jenkins-hbase4:42483] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,824 INFO [RS:2;jenkins-hbase4:42483] regionserver.LeaseManager(133): Closed leases 2023-07-20 04:05:53,825 INFO [RS:2;jenkins-hbase4:42483] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-20 04:05:53,825 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 04:05:53,826 INFO [RS:2;jenkins-hbase4:42483] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42483 2023-07-20 04:05:53,829 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42483,1689825934533 2023-07-20 04:05:53,829 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:53,829 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 04:05:53,831 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42483,1689825934533] 2023-07-20 04:05:53,831 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42483,1689825934533; numProcessing=1 2023-07-20 04:05:53,832 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42483,1689825934533 already deleted, retry=false 2023-07-20 04:05:53,832 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42483,1689825934533 expired; onlineServers=0 2023-07-20 04:05:53,832 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,34081,1689825928319' ***** 2023-07-20 04:05:53,832 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-20 04:05:53,833 DEBUG [M:0;jenkins-hbase4:34081] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50f708d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 04:05:53,833 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,34081,1689825928319 2023-07-20 04:05:53,833 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,34081,1689825928319; all regions closed. 2023-07-20 04:05:53,833 DEBUG [M:0;jenkins-hbase4:34081] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 04:05:53,833 DEBUG [M:0;jenkins-hbase4:34081] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-20 04:05:53,833 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-20 04:05:53,833 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689825930818] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689825930818,5,FailOnTimeoutGroup] 2023-07-20 04:05:53,833 DEBUG [M:0;jenkins-hbase4:34081] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-20 04:05:53,833 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689825930816] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689825930816,5,FailOnTimeoutGroup] 2023-07-20 04:05:53,834 INFO [M:0;jenkins-hbase4:34081] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-20 04:05:53,834 INFO [M:0;jenkins-hbase4:34081] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-20 04:05:53,834 INFO [M:0;jenkins-hbase4:34081] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-20 04:05:53,834 DEBUG [M:0;jenkins-hbase4:34081] master.HMaster(1512): Stopping service threads 2023-07-20 04:05:53,834 INFO [M:0;jenkins-hbase4:34081] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-20 04:05:53,835 ERROR [M:0;jenkins-hbase4:34081] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1195982799) connection to localhost/127.0.0.1:39169 from jenkins,5,PEWorkerGroup] 2023-07-20 04:05:53,835 INFO [M:0;jenkins-hbase4:34081] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-20 04:05:53,835 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-20 04:05:53,836 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-20 04:05:53,836 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 04:05:53,836 DEBUG [M:0;jenkins-hbase4:34081] zookeeper.ZKUtil(398): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-20 04:05:53,836 WARN [M:0;jenkins-hbase4:34081] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-20 04:05:53,836 INFO [M:0;jenkins-hbase4:34081] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-20 04:05:53,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 04:05:53,836 INFO [M:0;jenkins-hbase4:34081] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-20 04:05:53,837 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 04:05:53,837 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:53,837 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:53,837 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 04:05:53,837 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:53,837 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=96.33 KB heapSize=116.31 KB 2023-07-20 04:05:53,855 INFO [M:0;jenkins-hbase4:34081] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=96.33 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2bc313ff180642ad8ff27936680c2bbe 2023-07-20 04:05:53,862 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2bc313ff180642ad8ff27936680c2bbe as hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2bc313ff180642ad8ff27936680c2bbe 2023-07-20 04:05:53,868 INFO [M:0;jenkins-hbase4:34081] regionserver.HStore(1080): Added hdfs://localhost:39169/user/jenkins/test-data/5b2205cc-4b09-df88-a020-ab4186d57269/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2bc313ff180642ad8ff27936680c2bbe, entries=29, sequenceid=225, filesize=8.3 K 2023-07-20 04:05:53,869 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegion(2948): Finished flush of dataSize ~96.33 KB/98642, heapSize ~116.30 KB/119088, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 32ms, sequenceid=225, compaction requested=false 2023-07-20 04:05:53,870 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 04:05:53,870 DEBUG [M:0;jenkins-hbase4:34081] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 04:05:53,874 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 04:05:53,874 INFO [M:0;jenkins-hbase4:34081] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-20 04:05:53,874 INFO [M:0;jenkins-hbase4:34081] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34081 2023-07-20 04:05:53,879 DEBUG [M:0;jenkins-hbase4:34081] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,34081,1689825928319 already deleted, retry=false 2023-07-20 04:05:54,085 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:54,085 INFO [M:0;jenkins-hbase4:34081] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,34081,1689825928319; zookeeper connection closed. 2023-07-20 04:05:54,085 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): master:34081-0x10180e7e43c0000, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:54,185 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:54,185 INFO [RS:2;jenkins-hbase4:42483] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,42483,1689825934533; zookeeper connection closed. 2023-07-20 04:05:54,185 DEBUG [Listener at localhost/35413-EventThread] zookeeper.ZKWatcher(600): regionserver:42483-0x10180e7e43c0008, quorum=127.0.0.1:60043, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 04:05:54,185 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7588ce25] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7588ce25 2023-07-20 04:05:54,186 INFO [Listener at localhost/35413] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-20 04:05:54,186 WARN [Listener at localhost/35413] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 04:05:54,189 INFO [Listener at localhost/35413] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 04:05:54,287 WARN [BP-1949644524-172.31.14.131-1689825925221 heartbeating to localhost/127.0.0.1:39169] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1949644524-172.31.14.131-1689825925221 (Datanode Uuid ce15205f-9767-442d-8e21-8bb4260140d2) service to localhost/127.0.0.1:39169 2023-07-20 04:05:54,289 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data3/current/BP-1949644524-172.31.14.131-1689825925221] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 04:05:54,289 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data4/current/BP-1949644524-172.31.14.131-1689825925221] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 04:05:54,295 WARN [Listener at localhost/35413] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 04:05:54,302 INFO [Listener at localhost/35413] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 04:05:54,408 WARN [BP-1949644524-172.31.14.131-1689825925221 heartbeating to localhost/127.0.0.1:39169] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 04:05:54,408 WARN [BP-1949644524-172.31.14.131-1689825925221 heartbeating to localhost/127.0.0.1:39169] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1949644524-172.31.14.131-1689825925221 (Datanode Uuid 2885965a-a266-45eb-b8d9-d40c7b55938f) service to localhost/127.0.0.1:39169 2023-07-20 04:05:54,408 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data1/current/BP-1949644524-172.31.14.131-1689825925221] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 04:05:54,409 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/79558410-5358-5813-52c1-7e1c8c59726c/cluster_0ae2956c-2d37-beb5-f9ae-b3d24a1853a7/dfs/data/data2/current/BP-1949644524-172.31.14.131-1689825925221] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 04:05:54,448 INFO [Listener at localhost/35413] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 04:05:54,561 INFO [Listener at localhost/35413] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-20 04:05:54,601 INFO [Listener at localhost/35413] hbase.HBaseTestingUtility(1293): Minicluster is down