2023-07-20 07:05:10,968 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-20 07:05:11,200 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249 2023-07-20 07:05:11,213 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-20 07:05:11,213 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab, deleteOnExit=true 2023-07-20 07:05:11,214 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-20 07:05:11,214 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/test.cache.data in system properties and HBase conf 2023-07-20 07:05:11,214 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/hadoop.tmp.dir in system properties and HBase conf 2023-07-20 07:05:11,215 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/hadoop.log.dir in system properties and HBase conf 2023-07-20 07:05:11,215 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-20 07:05:11,216 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-20 07:05:11,216 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-20 07:05:11,325 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-20 07:05:11,713 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-20 07:05:11,717 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-20 07:05:11,717 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-20 07:05:11,718 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-20 07:05:11,718 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 07:05:11,718 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-20 07:05:11,718 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-20 07:05:11,719 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-20 07:05:11,719 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 07:05:11,720 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-20 07:05:11,720 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/nfs.dump.dir in system properties and HBase conf 2023-07-20 07:05:11,720 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/java.io.tmpdir in system properties and HBase conf 2023-07-20 07:05:11,721 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-20 07:05:11,721 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-20 07:05:11,722 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-20 07:05:12,231 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 07:05:12,235 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 07:05:12,520 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-20 07:05:12,668 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-20 07:05:12,682 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 07:05:12,718 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 07:05:12,748 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/java.io.tmpdir/Jetty_localhost_36567_hdfs____.uzv0tf/webapp 2023-07-20 07:05:12,883 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36567 2023-07-20 07:05:12,900 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-20 07:05:12,900 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-20 07:05:13,388 WARN [Listener at localhost/35421] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 07:05:13,454 WARN [Listener at localhost/35421] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 07:05:13,476 WARN [Listener at localhost/35421] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 07:05:13,482 INFO [Listener at localhost/35421] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 07:05:13,487 INFO [Listener at localhost/35421] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/java.io.tmpdir/Jetty_localhost_41825_datanode____l3vw05/webapp 2023-07-20 07:05:13,593 INFO [Listener at localhost/35421] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41825 2023-07-20 07:05:13,931 WARN [Listener at localhost/40717] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 07:05:13,944 WARN [Listener at localhost/40717] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-20 07:05:13,947 WARN [Listener at localhost/40717] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-20 07:05:13,949 INFO [Listener at localhost/40717] log.Slf4jLog(67): jetty-6.1.26 2023-07-20 07:05:13,955 INFO [Listener at localhost/40717] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/java.io.tmpdir/Jetty_localhost_33659_datanode____tjmhup/webapp 2023-07-20 07:05:14,058 INFO [Listener at localhost/40717] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33659 2023-07-20 07:05:14,066 WARN [Listener at localhost/32987] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-20 07:05:14,363 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8bd6d9017558213e: Processing first storage report for DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab from datanode 7cc6d6db-1fef-40cb-a5b5-fab91404c01b 2023-07-20 07:05:14,365 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8bd6d9017558213e: from storage DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab node DatanodeRegistration(127.0.0.1:34747, datanodeUuid=7cc6d6db-1fef-40cb-a5b5-fab91404c01b, infoPort=33307, infoSecurePort=0, ipcPort=32987, storageInfo=lv=-57;cid=testClusterID;nsid=116571330;c=1689836712308), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-20 07:05:14,365 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1c0b6b6e9e79d6ef: Processing first storage report for DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc from datanode d1e4ee74-d84d-48d9-9600-b82ac8fc5a3b 2023-07-20 07:05:14,366 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1c0b6b6e9e79d6ef: from storage DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc node DatanodeRegistration(127.0.0.1:38425, datanodeUuid=d1e4ee74-d84d-48d9-9600-b82ac8fc5a3b, infoPort=44649, infoSecurePort=0, ipcPort=40717, storageInfo=lv=-57;cid=testClusterID;nsid=116571330;c=1689836712308), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-07-20 07:05:14,366 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x8bd6d9017558213e: Processing first storage report for DS-85ecffbc-3c6b-497e-a758-dacbab6134eb from datanode 7cc6d6db-1fef-40cb-a5b5-fab91404c01b 2023-07-20 07:05:14,366 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x8bd6d9017558213e: from storage DS-85ecffbc-3c6b-497e-a758-dacbab6134eb node DatanodeRegistration(127.0.0.1:34747, datanodeUuid=7cc6d6db-1fef-40cb-a5b5-fab91404c01b, infoPort=33307, infoSecurePort=0, ipcPort=32987, storageInfo=lv=-57;cid=testClusterID;nsid=116571330;c=1689836712308), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 07:05:14,366 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1c0b6b6e9e79d6ef: Processing first storage report for DS-dea6aa0e-2799-4b58-bfaa-001f27807bd4 from datanode d1e4ee74-d84d-48d9-9600-b82ac8fc5a3b 2023-07-20 07:05:14,366 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1c0b6b6e9e79d6ef: from storage DS-dea6aa0e-2799-4b58-bfaa-001f27807bd4 node DatanodeRegistration(127.0.0.1:38425, datanodeUuid=d1e4ee74-d84d-48d9-9600-b82ac8fc5a3b, infoPort=44649, infoSecurePort=0, ipcPort=40717, storageInfo=lv=-57;cid=testClusterID;nsid=116571330;c=1689836712308), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-20 07:05:14,441 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249 2023-07-20 07:05:14,512 INFO [Listener at localhost/32987] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/zookeeper_0, clientPort=60197, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-20 07:05:14,525 INFO [Listener at localhost/32987] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60197 2023-07-20 07:05:14,535 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:14,538 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:15,201 INFO [Listener at localhost/32987] util.FSUtils(471): Created version file at hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1 with version=8 2023-07-20 07:05:15,201 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/hbase-staging 2023-07-20 07:05:15,518 INFO [Listener at localhost/32987] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-20 07:05:15,988 INFO [Listener at localhost/32987] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 07:05:16,020 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,021 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,021 INFO [Listener at localhost/32987] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 07:05:16,021 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,022 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 07:05:16,164 INFO [Listener at localhost/32987] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 07:05:16,235 DEBUG [Listener at localhost/32987] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-20 07:05:16,329 INFO [Listener at localhost/32987] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40953 2023-07-20 07:05:16,339 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,341 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,362 INFO [Listener at localhost/32987] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40953 connecting to ZooKeeper ensemble=127.0.0.1:60197 2023-07-20 07:05:16,402 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:409530x0, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 07:05:16,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:40953-0x101818c7d050000 connected 2023-07-20 07:05:16,429 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 07:05:16,430 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 07:05:16,434 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 07:05:16,441 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40953 2023-07-20 07:05:16,442 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40953 2023-07-20 07:05:16,442 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40953 2023-07-20 07:05:16,443 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40953 2023-07-20 07:05:16,443 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40953 2023-07-20 07:05:16,449 INFO [Listener at localhost/32987] master.HMaster(444): hbase.rootdir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1, hbase.cluster.distributed=false 2023-07-20 07:05:16,516 INFO [Listener at localhost/32987] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 07:05:16,516 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,516 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,516 INFO [Listener at localhost/32987] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 07:05:16,516 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,517 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 07:05:16,521 INFO [Listener at localhost/32987] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 07:05:16,524 INFO [Listener at localhost/32987] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33117 2023-07-20 07:05:16,526 INFO [Listener at localhost/32987] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 07:05:16,533 DEBUG [Listener at localhost/32987] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 07:05:16,534 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,536 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,537 INFO [Listener at localhost/32987] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33117 connecting to ZooKeeper ensemble=127.0.0.1:60197 2023-07-20 07:05:16,541 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:331170x0, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 07:05:16,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:33117-0x101818c7d050001 connected 2023-07-20 07:05:16,542 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 07:05:16,543 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 07:05:16,544 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 07:05:16,545 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33117 2023-07-20 07:05:16,545 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33117 2023-07-20 07:05:16,545 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33117 2023-07-20 07:05:16,546 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33117 2023-07-20 07:05:16,546 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33117 2023-07-20 07:05:16,559 INFO [Listener at localhost/32987] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 07:05:16,560 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,560 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,560 INFO [Listener at localhost/32987] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 07:05:16,560 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:16,561 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 07:05:16,561 INFO [Listener at localhost/32987] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 07:05:16,562 INFO [Listener at localhost/32987] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39933 2023-07-20 07:05:16,562 INFO [Listener at localhost/32987] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 07:05:16,564 DEBUG [Listener at localhost/32987] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 07:05:16,564 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,566 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,568 INFO [Listener at localhost/32987] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39933 connecting to ZooKeeper ensemble=127.0.0.1:60197 2023-07-20 07:05:16,571 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:399330x0, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 07:05:16,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39933-0x101818c7d050002 connected 2023-07-20 07:05:16,572 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 07:05:16,573 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 07:05:16,574 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 07:05:16,574 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39933 2023-07-20 07:05:16,575 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39933 2023-07-20 07:05:16,575 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39933 2023-07-20 07:05:16,576 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39933 2023-07-20 07:05:16,576 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39933 2023-07-20 07:05:16,577 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:16,587 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 07:05:16,588 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:16,608 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 07:05:16,608 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 07:05:16,608 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-20 07:05:16,608 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:16,610 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 07:05:16,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40953,1689836715351 from backup master directory 2023-07-20 07:05:16,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 07:05:16,614 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:16,614 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-20 07:05:16,615 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 07:05:16,615 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:16,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-20 07:05:16,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-20 07:05:16,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/hbase.id with ID: 65bacaa8-36d9-456b-9905-ec0f2285fed6 2023-07-20 07:05:16,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:16,759 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:16,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2689565b to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:16,834 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a864625, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:16,857 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 07:05:16,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-20 07:05:16,878 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-20 07:05:16,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-20 07:05:16,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:16,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:16,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:16,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store-tmp 2023-07-20 07:05:16,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:16,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 07:05:16,951 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:16,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:16,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 07:05:16,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:16,951 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:16,951 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 07:05:16,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/WALs/jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:16,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40953%2C1689836715351, suffix=, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/WALs/jenkins-hbase4.apache.org,40953,1689836715351, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/oldWALs, maxLogs=10 2023-07-20 07:05:17,028 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:17,028 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:17,036 DEBUG [RS-EventLoopGroup-4-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:17,098 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/WALs/jenkins-hbase4.apache.org,40953,1689836715351/jenkins-hbase4.apache.org%2C40953%2C1689836715351.1689836716983 2023-07-20 07:05:17,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK], DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK]] 2023-07-20 07:05:17,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:17,099 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:17,102 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,158 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,165 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-20 07:05:17,188 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-20 07:05:17,199 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:17,205 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,206 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-20 07:05:17,225 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 07:05:17,226 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66685829, jitterRate=-0.006303712725639343}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:17,226 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 07:05:17,227 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-20 07:05:17,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-20 07:05:17,247 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-20 07:05:17,249 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-20 07:05:17,250 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-20 07:05:17,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-07-20 07:05:17,283 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-20 07:05:17,307 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-20 07:05:17,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-20 07:05:17,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-20 07:05:17,342 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-20 07:05:17,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-20 07:05:17,348 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-20 07:05:17,352 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-20 07:05:17,355 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:17,356 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-20 07:05:17,357 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-20 07:05:17,368 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-20 07:05:17,373 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 07:05:17,373 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 07:05:17,373 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-20 07:05:17,373 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:17,374 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40953,1689836715351, sessionid=0x101818c7d050000, setting cluster-up flag (Was=false) 2023-07-20 07:05:17,389 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:17,393 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-20 07:05:17,395 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:17,400 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:17,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-20 07:05:17,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:17,411 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.hbase-snapshot/.tmp 2023-07-20 07:05:17,480 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(951): ClusterId : 65bacaa8-36d9-456b-9905-ec0f2285fed6 2023-07-20 07:05:17,480 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(951): ClusterId : 65bacaa8-36d9-456b-9905-ec0f2285fed6 2023-07-20 07:05:17,485 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 07:05:17,485 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 07:05:17,491 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 07:05:17,491 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 07:05:17,491 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 07:05:17,491 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 07:05:17,496 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 07:05:17,496 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 07:05:17,497 DEBUG [RS:1;jenkins-hbase4:39933] zookeeper.ReadOnlyZKClient(139): Connect 0x3c60ef52 to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:17,498 DEBUG [RS:0;jenkins-hbase4:33117] zookeeper.ReadOnlyZKClient(139): Connect 0x3f4290a1 to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:17,510 DEBUG [RS:1;jenkins-hbase4:39933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32a706ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:17,510 DEBUG [RS:0;jenkins-hbase4:33117] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@31856939, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:17,511 DEBUG [RS:1;jenkins-hbase4:39933] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36988422, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 07:05:17,511 DEBUG [RS:0;jenkins-hbase4:33117] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35bafbd1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 07:05:17,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,536 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 07:05:17,537 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,538 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:33117 2023-07-20 07:05:17,538 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39933 2023-07-20 07:05:17,539 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689836747539 2023-07-20 07:05:17,542 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-20 07:05:17,542 INFO [RS:0;jenkins-hbase4:33117] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 07:05:17,543 INFO [RS:0;jenkins-hbase4:33117] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 07:05:17,542 INFO [RS:1;jenkins-hbase4:39933] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 07:05:17,543 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 07:05:17,543 INFO [RS:1;jenkins-hbase4:39933] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 07:05:17,543 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 07:05:17,546 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40953,1689836715351 with isa=jenkins-hbase4.apache.org/172.31.14.131:39933, startcode=1689836716559 2023-07-20 07:05:17,546 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40953,1689836715351 with isa=jenkins-hbase4.apache.org/172.31.14.131:33117, startcode=1689836716515 2023-07-20 07:05:17,546 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 07:05:17,547 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-20 07:05:17,552 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 07:05:17,552 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-20 07:05:17,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-20 07:05:17,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-20 07:05:17,560 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-20 07:05:17,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-20 07:05:17,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,562 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-20 07:05:17,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-20 07:05:17,564 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-20 07:05:17,564 DEBUG [RS:0;jenkins-hbase4:33117] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 07:05:17,564 DEBUG [RS:1;jenkins-hbase4:39933] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 07:05:17,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-20 07:05:17,568 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-20 07:05:17,570 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689836717569,5,FailOnTimeoutGroup] 2023-07-20 07:05:17,570 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689836717570,5,FailOnTimeoutGroup] 2023-07-20 07:05:17,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,570 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-20 07:05:17,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,592 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 07:05:17,594 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-20 07:05:17,594 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1 2023-07-20 07:05:17,631 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51883, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 07:05:17,631 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34467, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 07:05:17,644 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40953] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,660 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:17,663 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 07:05:17,665 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40953] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,670 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:17,671 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 07:05:17,672 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1 2023-07-20 07:05:17,672 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1 2023-07-20 07:05:17,672 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35421 2023-07-20 07:05:17,672 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35421 2023-07-20 07:05:17,672 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 07:05:17,672 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 07:05:17,673 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:17,673 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 07:05:17,676 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:17,678 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 07:05:17,679 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:17,679 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:17,680 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 07:05:17,684 DEBUG [RS:0;jenkins-hbase4:33117] zookeeper.ZKUtil(162): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,684 WARN [RS:0;jenkins-hbase4:33117] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 07:05:17,684 DEBUG [RS:1;jenkins-hbase4:39933] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,685 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:17,685 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39933,1689836716559] 2023-07-20 07:05:17,685 WARN [RS:1;jenkins-hbase4:39933] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 07:05:17,684 INFO [RS:0;jenkins-hbase4:33117] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:17,686 INFO [RS:1;jenkins-hbase4:39933] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:17,686 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33117,1689836716515] 2023-07-20 07:05:17,686 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,686 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,687 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 07:05:17,688 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:17,694 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:17,696 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:17,705 DEBUG [RS:0;jenkins-hbase4:33117] zookeeper.ZKUtil(162): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,705 DEBUG [RS:1;jenkins-hbase4:39933] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,706 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 07:05:17,706 DEBUG [RS:0;jenkins-hbase4:33117] zookeeper.ZKUtil(162): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,707 DEBUG [RS:1;jenkins-hbase4:39933] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,711 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 07:05:17,721 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 07:05:17,721 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 07:05:17,722 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 07:05:17,726 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62612354, jitterRate=-0.06700322031974792}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 07:05:17,726 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 07:05:17,726 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 07:05:17,726 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 07:05:17,726 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 07:05:17,727 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 07:05:17,727 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 07:05:17,731 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:17,731 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 07:05:17,733 INFO [RS:0;jenkins-hbase4:33117] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 07:05:17,733 INFO [RS:1;jenkins-hbase4:39933] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 07:05:17,738 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-20 07:05:17,738 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-20 07:05:17,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 07:05:17,763 INFO [RS:0;jenkins-hbase4:33117] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 07:05:17,763 INFO [RS:1;jenkins-hbase4:39933] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 07:05:17,768 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 07:05:17,771 INFO [RS:1;jenkins-hbase4:39933] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 07:05:17,771 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,772 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 07:05:17,772 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-20 07:05:17,771 INFO [RS:0;jenkins-hbase4:33117] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 07:05:17,773 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,778 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 07:05:17,782 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,782 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,782 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,783 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 07:05:17,784 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 07:05:17,784 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,784 DEBUG [RS:1;jenkins-hbase4:39933] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,785 DEBUG [RS:0;jenkins-hbase4:33117] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:17,790 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,790 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,790 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,790 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,790 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,790 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,810 INFO [RS:0;jenkins-hbase4:33117] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 07:05:17,810 INFO [RS:1;jenkins-hbase4:39933] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 07:05:17,813 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33117,1689836716515-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,813 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39933,1689836716559-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:17,834 INFO [RS:1;jenkins-hbase4:39933] regionserver.Replication(203): jenkins-hbase4.apache.org,39933,1689836716559 started 2023-07-20 07:05:17,835 INFO [RS:0;jenkins-hbase4:33117] regionserver.Replication(203): jenkins-hbase4.apache.org,33117,1689836716515 started 2023-07-20 07:05:17,835 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39933,1689836716559, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39933, sessionid=0x101818c7d050002 2023-07-20 07:05:17,835 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,33117,1689836716515, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33117, sessionid=0x101818c7d050001 2023-07-20 07:05:17,835 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 07:05:17,835 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 07:05:17,835 DEBUG [RS:1;jenkins-hbase4:39933] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,835 DEBUG [RS:0;jenkins-hbase4:33117] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,836 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39933,1689836716559' 2023-07-20 07:05:17,836 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33117,1689836716515' 2023-07-20 07:05:17,837 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 07:05:17,837 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 07:05:17,838 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 07:05:17,838 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 07:05:17,838 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 07:05:17,838 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 07:05:17,838 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 07:05:17,839 DEBUG [RS:0;jenkins-hbase4:33117] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:17,839 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 07:05:17,839 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33117,1689836716515' 2023-07-20 07:05:17,839 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 07:05:17,839 DEBUG [RS:1;jenkins-hbase4:39933] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:17,839 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39933,1689836716559' 2023-07-20 07:05:17,840 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 07:05:17,840 DEBUG [RS:0;jenkins-hbase4:33117] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 07:05:17,841 DEBUG [RS:1;jenkins-hbase4:39933] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 07:05:17,842 DEBUG [RS:0;jenkins-hbase4:33117] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 07:05:17,842 INFO [RS:0;jenkins-hbase4:33117] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 07:05:17,842 INFO [RS:0;jenkins-hbase4:33117] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 07:05:17,842 DEBUG [RS:1;jenkins-hbase4:39933] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 07:05:17,842 INFO [RS:1;jenkins-hbase4:39933] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 07:05:17,843 INFO [RS:1;jenkins-hbase4:39933] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 07:05:17,925 DEBUG [jenkins-hbase4:40953] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-20 07:05:17,930 DEBUG [jenkins-hbase4:40953] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 07:05:17,938 DEBUG [jenkins-hbase4:40953] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 07:05:17,938 DEBUG [jenkins-hbase4:40953] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 07:05:17,938 DEBUG [jenkins-hbase4:40953] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 07:05:17,941 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33117,1689836716515, state=OPENING 2023-07-20 07:05:17,948 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-20 07:05:17,950 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:17,951 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:17,953 INFO [RS:0;jenkins-hbase4:33117] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33117%2C1689836716515, suffix=, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:17,954 INFO [RS:1;jenkins-hbase4:39933] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39933%2C1689836716559, suffix=, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:17,956 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33117,1689836716515}] 2023-07-20 07:05:17,976 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:17,976 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:17,990 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:17,990 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:17,991 INFO [RS:0;jenkins-hbase4:33117] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 2023-07-20 07:05:17,995 DEBUG [RS:0;jenkins-hbase4:33117] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK], DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK]] 2023-07-20 07:05:17,997 INFO [RS:1;jenkins-hbase4:39933] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 2023-07-20 07:05:17,997 DEBUG [RS:1;jenkins-hbase4:39933] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK]] 2023-07-20 07:05:18,145 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:18,147 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 07:05:18,150 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60096, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 07:05:18,163 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 07:05:18,163 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:18,166 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta, suffix=.meta, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:18,184 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:18,184 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:18,193 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta 2023-07-20 07:05:18,194 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK]] 2023-07-20 07:05:18,195 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:18,196 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 07:05:18,211 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 07:05:18,246 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 07:05:18,251 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 07:05:18,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:18,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 07:05:18,252 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 07:05:18,255 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 07:05:18,257 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:18,257 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:18,257 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 07:05:18,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:18,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 07:05:18,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:18,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:18,261 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 07:05:18,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:18,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 07:05:18,264 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:18,264 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:18,264 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 07:05:18,265 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:18,267 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:18,270 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:18,273 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 07:05:18,276 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 07:05:18,277 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60195465, jitterRate=-0.10301767289638519}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 07:05:18,277 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 07:05:18,286 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689836718137 2023-07-20 07:05:18,303 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 07:05:18,303 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 07:05:18,304 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,33117,1689836716515, state=OPEN 2023-07-20 07:05:18,307 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 07:05:18,307 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:18,311 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-20 07:05:18,311 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,33117,1689836716515 in 351 msec 2023-07-20 07:05:18,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-20 07:05:18,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 563 msec 2023-07-20 07:05:18,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 866 msec 2023-07-20 07:05:18,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689836718321, completionTime=-1 2023-07-20 07:05:18,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-20 07:05:18,322 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-20 07:05:18,391 DEBUG [hconnection-0x3943119-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:18,394 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:18,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-20 07:05:18,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689836778410 2023-07-20 07:05:18,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689836838410 2023-07-20 07:05:18,410 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-07-20 07:05:18,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40953,1689836715351-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:18,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40953,1689836715351-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:18,432 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40953,1689836715351-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:18,434 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40953, period=300000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:18,434 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:18,441 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-20 07:05:18,449 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-20 07:05:18,450 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-20 07:05:18,459 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-20 07:05:18,461 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 07:05:18,464 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 07:05:18,487 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,489 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 empty. 2023-07-20 07:05:18,490 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,490 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-20 07:05:18,516 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-20 07:05:18,518 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6707c3fc6807504cee3bc124f1adff97, NAME => 'hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp 2023-07-20 07:05:18,534 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:18,535 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 6707c3fc6807504cee3bc124f1adff97, disabling compactions & flushes 2023-07-20 07:05:18,535 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,535 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,535 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. after waiting 0 ms 2023-07-20 07:05:18,535 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,535 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,535 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:18,539 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 07:05:18,557 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836718541"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689836718541"}]},"ts":"1689836718541"} 2023-07-20 07:05:18,582 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 07:05:18,584 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 07:05:18,588 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689836718584"}]},"ts":"1689836718584"} 2023-07-20 07:05:18,593 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-20 07:05:18,597 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 07:05:18,598 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 07:05:18,598 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 07:05:18,599 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 07:05:18,601 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN}] 2023-07-20 07:05:18,603 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN 2023-07-20 07:05:18,606 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33117,1689836716515; forceNewPlan=false, retain=false 2023-07-20 07:05:18,758 INFO [jenkins-hbase4:40953] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 07:05:18,759 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:18,759 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836718758"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836718758"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836718758"}]},"ts":"1689836718758"} 2023-07-20 07:05:18,763 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,33117,1689836716515}] 2023-07-20 07:05:18,923 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,923 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6707c3fc6807504cee3bc124f1adff97, NAME => 'hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:18,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:18,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,924 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,927 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,929 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:18,929 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:18,930 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6707c3fc6807504cee3bc124f1adff97 columnFamilyName info 2023-07-20 07:05:18,930 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(310): Store=6707c3fc6807504cee3bc124f1adff97/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:18,932 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,933 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,937 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:18,940 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 07:05:18,941 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6707c3fc6807504cee3bc124f1adff97; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65024407, jitterRate=-0.031060829758644104}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:18,942 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:18,944 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97., pid=6, masterSystemTime=1689836718916 2023-07-20 07:05:18,947 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,948 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:18,949 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:18,949 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836718948"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836718948"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836718948"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836718948"}]},"ts":"1689836718948"} 2023-07-20 07:05:18,957 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-20 07:05:18,957 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,33117,1689836716515 in 190 msec 2023-07-20 07:05:18,961 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-20 07:05:18,961 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN in 356 msec 2023-07-20 07:05:18,962 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 07:05:18,963 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689836718963"}]},"ts":"1689836718963"} 2023-07-20 07:05:18,966 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-20 07:05:18,969 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 07:05:18,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 519 msec 2023-07-20 07:05:19,062 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-20 07:05:19,063 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-20 07:05:19,063 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:19,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-20 07:05:19,113 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 07:05:19,118 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-07-20 07:05:19,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-20 07:05:19,141 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-20 07:05:19,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-20 07:05:19,155 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-20 07:05:19,159 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-20 07:05:19,159 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.544sec 2023-07-20 07:05:19,162 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-20 07:05:19,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-20 07:05:19,164 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-20 07:05:19,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40953,1689836715351-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-20 07:05:19,165 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40953,1689836715351-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-20 07:05:19,174 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-20 07:05:19,186 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(139): Connect 0x5363c3cc to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:19,191 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@aa7b11b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:19,204 DEBUG [hconnection-0x31add453-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:19,215 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60106, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:19,224 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:19,236 DEBUG [Listener at localhost/32987] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-20 07:05:19,239 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43934, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-20 07:05:19,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-20 07:05:19,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-20 07:05:19,256 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-20 07:05:19,257 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-20 07:05:19,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.MasterRpcServices(700): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-20 07:05:19,260 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,261 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/default/t/0b729b0d456def9e430b606c44f89497 empty. 2023-07-20 07:05:19,263 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,263 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-20 07:05:19,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 07:05:19,283 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-20 07:05:19,284 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0b729b0d456def9e430b606c44f89497, NAME => 't,,1689836719248.0b729b0d456def9e430b606c44f89497.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/.tmp 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689836719248.0b729b0d456def9e430b606c44f89497.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing 0b729b0d456def9e430b606c44f89497, disabling compactions & flushes 2023-07-20 07:05:19,300 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. after waiting 0 ms 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,300 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,300 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:19,304 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-20 07:05:19,306 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836719306"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689836719306"}]},"ts":"1689836719306"} 2023-07-20 07:05:19,309 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-20 07:05:19,310 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-20 07:05:19,310 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689836719310"}]},"ts":"1689836719310"} 2023-07-20 07:05:19,312 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-20 07:05:19,317 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-07-20 07:05:19,318 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-20 07:05:19,318 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-20 07:05:19,318 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-20 07:05:19,318 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN}] 2023-07-20 07:05:19,320 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN 2023-07-20 07:05:19,321 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33117,1689836716515; forceNewPlan=false, retain=false 2023-07-20 07:05:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 07:05:19,472 INFO [jenkins-hbase4:40953] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-20 07:05:19,472 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:19,473 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836719472"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836719472"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836719472"}]},"ts":"1689836719472"} 2023-07-20 07:05:19,476 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,33117,1689836716515}] 2023-07-20 07:05:19,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 07:05:19,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b729b0d456def9e430b606c44f89497, NAME => 't,,1689836719248.0b729b0d456def9e430b606c44f89497.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:19,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689836719248.0b729b0d456def9e430b606c44f89497.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:19,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,638 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,640 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:19,640 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:19,641 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b729b0d456def9e430b606c44f89497 columnFamilyName f 2023-07-20 07:05:19,641 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(310): Store=0b729b0d456def9e430b606c44f89497/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:19,643 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,644 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,648 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,651 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-20 07:05:19,652 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b729b0d456def9e430b606c44f89497; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70095837, jitterRate=0.044509366154670715}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:19,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:19,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689836719248.0b729b0d456def9e430b606c44f89497., pid=11, masterSystemTime=1689836719629 2023-07-20 07:05:19,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,657 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:19,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836719657"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836719657"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836719657"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836719657"}]},"ts":"1689836719657"} 2023-07-20 07:05:19,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-20 07:05:19,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,33117,1689836716515 in 184 msec 2023-07-20 07:05:19,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-20 07:05:19,666 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN in 345 msec 2023-07-20 07:05:19,667 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-20 07:05:19,667 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689836719667"}]},"ts":"1689836719667"} 2023-07-20 07:05:19,669 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-20 07:05:19,672 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-20 07:05:19,674 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 420 msec 2023-07-20 07:05:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-20 07:05:19,878 INFO [Listener at localhost/32987] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-20 07:05:19,878 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-20 07:05:19,880 INFO [Listener at localhost/32987] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 07:05:19,885 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-20 07:05:19,885 INFO [Listener at localhost/32987] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-20 07:05:19,886 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-20 07:05:19,930 INFO [Listener at localhost/32987] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=294, OpenFileDescriptor=581, MaxFileDescriptor=60000, SystemLoadAverage=139, ProcessCount=170, AvailableMemoryMB=5786 2023-07-20 07:05:19,931 INFO [Listener at localhost/32987] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-20 07:05:19,935 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 07:05:19,935 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,936 INFO [Listener at localhost/32987] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-20 07:05:19,936 INFO [Listener at localhost/32987] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-20 07:05:19,939 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689836718963/Put/vlen=2/seqid=0} 2023-07-20 07:05:19,939 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-20 07:05:19,939 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:regioninfo/1689836718948/Put/vlen=41/seqid=0, hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:seqnumDuringOpen/1689836718948/Put/vlen=8/seqid=0, hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:server/1689836718948/Put/vlen=31/seqid=0, hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:serverstartcode/1689836718948/Put/vlen=8/seqid=0, hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:sn/1689836718758/Put/vlen=45/seqid=0, hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97./info:state/1689836718948/Put/vlen=4/seqid=0} 2023-07-20 07:05:19,939 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 6707c3fc6807504cee3bc124f1adff97, NAME => 'hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:19,940 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689836719667/Put/vlen=2/seqid=0} 2023-07-20 07:05:19,940 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-20 07:05:19,940 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:regioninfo/1689836719657/Put/vlen=35/seqid=0, t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:seqnumDuringOpen/1689836719657/Put/vlen=8/seqid=0, t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:server/1689836719657/Put/vlen=31/seqid=0, t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:serverstartcode/1689836719657/Put/vlen=8/seqid=0, t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:sn/1689836719472/Put/vlen=45/seqid=0, t,,1689836719248.0b729b0d456def9e430b606c44f89497./info:state/1689836719657/Put/vlen=4/seqid=0} 2023-07-20 07:05:19,940 INFO [Listener at localhost/32987] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => 0b729b0d456def9e430b606c44f89497, NAME => 't,,1689836719248.0b729b0d456def9e430b606c44f89497.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:19,941 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(139): Connect 0x1ef800db to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:19,946 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1558c70b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:19,946 INFO [Listener at localhost/32987] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-20 07:05:19,946 INFO [Listener at localhost/32987] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-20 07:05:19,948 DEBUG [hconnection-0x155a17a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:19,951 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60110, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:19,956 INFO [Listener at localhost/32987] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-20 07:05:19,959 ERROR [Listener at localhost/32987] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,33117,1689836716515: die! ***** 2023-07-20 07:05:19,960 ERROR [Listener at localhost/32987] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 07:05:19,963 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 07:05:19,965 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 07:05:19,966 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 07:05:19,966 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 07:05:19,967 INFO [Listener at localhost/32987] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1001914368, "init": 513802240, "max": 2051014656, "used": 431545128 }, "NonHeapMemoryUsage": { "committed": 113139712, "init": 2555904, "max": -1, "used": 111093880 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org" } ] } 2023-07-20 07:05:19,970 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40953] master.MasterRpcServices(612): jenkins-hbase4.apache.org,33117,1689836716515 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,33117,1689836716515: die! ***** 2023-07-20 07:05:19,972 INFO [Listener at localhost/32987] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,33117,1689836716515' ***** 2023-07-20 07:05:19,973 INFO [Listener at localhost/32987] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 07:05:19,973 INFO [RS:0;jenkins-hbase4:33117] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 07:05:19,973 INFO [RS:0;jenkins-hbase4:33117] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 07:05:19,973 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 07:05:19,973 INFO [RS:0;jenkins-hbase4:33117] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 07:05:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33117] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner 5865075866173702156 2023-07-20 07:05:19,974 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(3305): Received CLOSE for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33117] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 07:05:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33117] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 172.31.14.131:60110 deadline: 1689836779973, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting 2023-07-20 07:05:19,974 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(3305): Received CLOSE for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:19,975 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:19,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b729b0d456def9e430b606c44f89497, disabling compactions & flushes 2023-07-20 07:05:19,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,976 DEBUG [RS:0;jenkins-hbase4:33117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f4290a1 to 127.0.0.1:60197 2023-07-20 07:05:19,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,976 DEBUG [RS:0;jenkins-hbase4:33117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:19,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. after waiting 0 ms 2023-07-20 07:05:19,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,976 INFO [RS:0;jenkins-hbase4:33117] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 07:05:19,976 INFO [RS:0;jenkins-hbase4:33117] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 07:05:19,976 INFO [RS:0;jenkins-hbase4:33117] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 07:05:19,976 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 07:05:19,976 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,977 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 07:05:19,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,978 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1478): Online Regions={0b729b0d456def9e430b606c44f89497=t,,1689836719248.0b729b0d456def9e430b606c44f89497., 1588230740=hbase:meta,,1.1588230740, 6707c3fc6807504cee3bc124f1adff97=hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.} 2023-07-20 07:05:19,978 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 07:05:19,978 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:19,979 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 07:05:19,979 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 07:05:19,979 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 07:05:19,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:19,979 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 07:05:19,979 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6707c3fc6807504cee3bc124f1adff97, disabling compactions & flushes 2023-07-20 07:05:19,979 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. after waiting 0 ms 2023-07-20 07:05:19,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,980 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,981 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,981 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:19,981 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 2394 in region hbase:meta,,1.1588230740 2023-07-20 07:05:19,981 DEBUG [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1504): Waiting on 1588230740, 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:19,981 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 07:05:19,982 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:19,982 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:19,982 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 07:05:19,982 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:19,995 DEBUG [Listener at localhost/32987] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 07:05:19,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33117] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 07:05:19,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33117] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 172.31.14.131:60110 deadline: 1689836779997, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting 2023-07-20 07:05:20,101 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33117] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2513) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44992) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 07:05:20,102 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33117] ipc.CallRunner(144): callId: 4 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:60110 deadline: 1689836780101, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,33117,1689836716515 aborting 2023-07-20 07:05:20,182 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,33117,1689836716515; all regions closed. 2023-07-20 07:05:20,196 DEBUG [RS:0;jenkins-hbase4:33117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:20,197 INFO [RS:0;jenkins-hbase4:33117] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:20,197 INFO [RS:0;jenkins-hbase4:33117] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-20 07:05:20,197 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 07:05:20,198 INFO [RS:0;jenkins-hbase4:33117] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33117 2023-07-20 07:05:20,206 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33117 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:20,207 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33117 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 2023-07-20 07:05:20,208 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:20,208 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:20,208 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:20,208 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:20,208 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:20,210 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33117,1689836716515] 2023-07-20 07:05:20,210 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33117,1689836716515; numProcessing=1 2023-07-20 07:05:20,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,211 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,33117,1689836716515 znode expired, triggering replicatorRemoved event 2023-07-20 07:05:20,212 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33117,1689836716515 already deleted, retry=false 2023-07-20 07:05:20,212 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,33117,1689836716515 on jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:20,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,219 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,33117,1689836716515, splitWal=true, meta=true 2023-07-20 07:05:20,220 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase4.apache.org,33117,1689836716515 (carryingMeta=true) jenkins-hbase4.apache.org,33117,1689836716515/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@4b175098[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 07:05:20,222 INFO [PEWorker-4] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33117,1689836716515, splitWal=true, meta=true 2023-07-20 07:05:20,224 INFO [PEWorker-4] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33117,1689836716515, splitWal=true, meta=true, isMeta: true 2023-07-20 07:05:20,226 DEBUG [PEWorker-4] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting 2023-07-20 07:05:20,227 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,33117,1689836716515 WAL count=1, meta=true 2023-07-20 07:05:20,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta}] 2023-07-20 07:05:20,238 DEBUG [PEWorker-5] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,239 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, worker=jenkins-hbase4.apache.org,39933,1689836716559}] 2023-07-20 07:05:20,310 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:20,311 INFO [RS:0;jenkins-hbase4:33117] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,33117,1689836716515; zookeeper connection closed. 2023-07-20 07:05:20,311 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:33117-0x101818c7d050001, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:20,311 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@79084577] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@79084577 2023-07-20 07:05:20,403 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,404 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 07:05:20,407 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 07:05:20,409 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39933] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-20 07:05:20,410 DEBUG [hconnection-0x155a17a4-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:33117 this server is in the failed servers list 2023-07-20 07:05:20,435 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, size=3.1 K (3185bytes) 2023-07-20 07:05:20,436 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta 2023-07-20 07:05:20,441 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta after 5ms 2023-07-20 07:05:20,447 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta took 13ms 2023-07-20 07:05:20,466 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-20 07:05:20,468 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta so closing down 2023-07-20 07:05:20,469 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 07:05:20,474 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-20 07:05:20,477 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp 2023-07-20 07:05:20,479 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 07:05:20,500 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 07:05:20,502 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 07:05:20,503 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 53 ms; skipped=1; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false 2023-07-20 07:05:20,503 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, journal: Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, size=3.1 K (3185bytes) at 1689836720436Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta so closing down at 1689836720468 (+32 ms)Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp at 1689836720477 (+9 ms)3 split writer threads finished at 1689836720479 (+2 ms)Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689836720500 (+21 ms)Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 at 1689836720502 (+2 ms)Processed 11 edits across 1 Regions in 53 ms; skipped=1; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, size=3.1 K, length=3185, corrupted=false, cancelled=false at 1689836720503 (+1 ms) 2023-07-20 07:05:20,503 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta 2023-07-20 07:05:20,506 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-20 07:05:20,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.HMaster(3641): Remote procedure done, pid=14 2023-07-20 07:05:20,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] wal.WALSplitUtil(141): Moved hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:20,523 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-20 07:05:20,523 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, worker=jenkins-hbase4.apache.org,39933,1689836716559 in 278 msec 2023-07-20 07:05:20,525 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,529 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-20 07:05:20,530 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.meta.1689836718169.meta, worker=jenkins-hbase4.apache.org,39933,1689836716559 in 297 msec 2023-07-20 07:05:20,532 INFO [PEWorker-4] master.SplitLogManager(171): hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting dir is empty, no logs to split. 2023-07-20 07:05:20,532 INFO [PEWorker-4] master.SplitWALManager(106): jenkins-hbase4.apache.org,33117,1689836716515 WAL count=0, meta=true 2023-07-20 07:05:20,533 DEBUG [PEWorker-4] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33117,1689836716515 WAL splitting is done? wals=0, meta=true 2023-07-20 07:05:20,534 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 07:05:20,536 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 07:05:20,537 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:20,688 DEBUG [jenkins-hbase4:40953] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 07:05:20,688 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39933,1689836716559, state=OPENING 2023-07-20 07:05:20,690 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 07:05:20,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39933,1689836716559}] 2023-07-20 07:05:20,690 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:20,712 WARN [ReadOnlyZKClient-127.0.0.1:60197@0x1ef800db] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 07:05:20,713 DEBUG [hconnection-0x155a17a4-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:20,716 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:20,716 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39933] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:48436 deadline: 1689836780716, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:20,795 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:20,848 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 07:05:20,849 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:20,851 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta, suffix=.meta, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:20,871 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:20,871 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:20,874 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta 2023-07-20 07:05:20,874 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK], DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK]] 2023-07-20 07:05:20,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:20,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 07:05:20,875 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 07:05:20,876 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 07:05:20,876 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 07:05:20,876 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:20,876 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 07:05:20,876 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 07:05:20,879 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 07:05:20,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:20,880 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:20,881 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 07:05:20,882 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:20,882 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 07:05:20,883 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:20,883 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:20,884 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 07:05:20,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:20,885 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 07:05:20,887 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:20,887 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:20,887 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 07:05:20,888 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:20,890 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:20,893 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:20,894 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 07:05:20,901 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 24, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 07:05:20,902 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=2.34 KB heapSize=4.86 KB 2023-07-20 07:05:20,953 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.18 KB at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/2dab5b285a774c6c95a9da15d4b2a801 2023-07-20 07:05:21,220 WARN [ReadOnlyZKClient-127.0.0.1:60197@0x1ef800db] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 07:05:21,221 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39933] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Get size: 76 connection: 172.31.14.131:48436 deadline: 1689836781221, exception=org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region hbase:meta,,1 is opening on jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:21,431 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/table/e33f477895ff430fa59b695507099b94 2023-07-20 07:05:21,444 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/2dab5b285a774c6c95a9da15d4b2a801 as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/2dab5b285a774c6c95a9da15d4b2a801 2023-07-20 07:05:21,456 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/2dab5b285a774c6c95a9da15d4b2a801, entries=20, sequenceid=13, filesize=6.9 K 2023-07-20 07:05:21,459 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/table/e33f477895ff430fa59b695507099b94 as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table/e33f477895ff430fa59b695507099b94 2023-07-20 07:05:21,482 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table/e33f477895ff430fa59b695507099b94, entries=4, sequenceid=13, filesize=4.7 K 2023-07-20 07:05:21,482 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~2.34 KB/2394, heapSize ~4.58 KB/4688, currentSize=0 B/0 for 1588230740 in 580ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 07:05:21,485 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-20 07:05:21,490 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2023-07-20 07:05:21,495 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 07:05:21,499 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 07:05:21,503 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 07:05:21,505 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70283070, jitterRate=0.047299355268478394}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 07:05:21,505 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 07:05:21,507 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=16, masterSystemTime=1689836720843 2023-07-20 07:05:21,510 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 07:05:21,511 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 07:05:21,512 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39933,1689836716559, state=OPEN 2023-07-20 07:05:21,514 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 07:05:21,514 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:21,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-20 07:05:21,518 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39933,1689836716559 in 824 msec 2023-07-20 07:05:21,522 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-20 07:05:21,522 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 984 msec 2023-07-20 07:05:21,523 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,33117,1689836716515 had 3 regions 2023-07-20 07:05:21,524 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,33117,1689836716515, splitWal=true, meta=true, isMeta: false 2023-07-20 07:05:21,528 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,33117,1689836716515 WAL count=1, meta=false 2023-07-20 07:05:21,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957}] 2023-07-20 07:05:21,532 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:21,533 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, worker=jenkins-hbase4.apache.org,39933,1689836716559}] 2023-07-20 07:05:21,687 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39933] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=18 2023-07-20 07:05:21,701 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, size=1.6 K (1665bytes) 2023-07-20 07:05:21,701 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 2023-07-20 07:05:21,702 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 after 1ms 2023-07-20 07:05:21,705 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 took 4ms 2023-07-20 07:05:21,709 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 6707c3fc6807504cee3bc124f1adff97: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-20 07:05:21,712 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 0b729b0d456def9e430b606c44f89497: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 18446744073709551615 } 2023-07-20 07:05:21,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 so closing down 2023-07-20 07:05:21,713 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 07:05:21,714 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-1] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp 2023-07-20 07:05:21,716 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp 2023-07-20 07:05:21,716 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 07:05:21,736 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-20 07:05:21,736 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-20 07:05:21,738 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000013 2023-07-20 07:05:21,738 INFO [split-log-closeStream-pool-1] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000005 2023-07-20 07:05:21,738 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 14 edits across 2 Regions in 32 ms; skipped=2; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, size=1.6 K, length=1665, corrupted=false, cancelled=false 2023-07-20 07:05:21,739 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, journal: Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, size=1.6 K (1665bytes) at 1689836721701Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 so closing down at 1689836721713 (+12 ms)Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp at 1689836721714 (+1 ms)Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp at 1689836721716 (+2 ms)3 split writer threads finished at 1689836721716Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689836721736 (+20 ms)Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689836721736Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000013 at 1689836721738 (+2 ms)Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000004-jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000005 at 1689836721738Processed 14 edits across 2 Regions in 32 ms; skipped=2; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, size=1.6 K, length=1665, corrupted=false, cancelled=false at 1689836721739 (+1 ms) 2023-07-20 07:05:21,739 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 2023-07-20 07:05:21,739 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=18 2023-07-20 07:05:21,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.HMaster(3641): Remote procedure done, pid=18 2023-07-20 07:05:21,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] wal.WALSplitUtil(141): Moved hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting/jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957 to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:21,747 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-20 07:05:21,748 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, worker=jenkins-hbase4.apache.org,39933,1689836716559 in 212 msec 2023-07-20 07:05:21,749 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:21,752 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=12 2023-07-20 07:05:21,752 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C33117%2C1689836716515.1689836717957, worker=jenkins-hbase4.apache.org,39933,1689836716559 in 221 msec 2023-07-20 07:05:21,754 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,33117,1689836716515-splitting dir is empty, no logs to split. 2023-07-20 07:05:21,754 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase4.apache.org,33117,1689836716515 WAL count=0, meta=false 2023-07-20 07:05:21,754 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,33117,1689836716515 WAL splitting is done? wals=0, meta=false 2023-07-20 07:05:21,757 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN}, {pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN}] 2023-07-20 07:05:21,758 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN 2023-07-20 07:05:21,758 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN 2023-07-20 07:05:21,759 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:21,759 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=19, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:21,909 DEBUG [jenkins-hbase4:40953] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 07:05:21,910 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:21,910 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:21,910 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836721910"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836721910"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836721910"}]},"ts":"1689836721910"} 2023-07-20 07:05:21,910 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836721910"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836721910"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836721910"}]},"ts":"1689836721910"} 2023-07-20 07:05:21,911 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33117 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:21,913 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33117 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 2023-07-20 07:05:22,015 DEBUG [PEWorker-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:22,017 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48440, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:22,020 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=19, state=RUNNABLE; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,39933,1689836716559}] 2023-07-20 07:05:22,021 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=20, state=RUNNABLE; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,39933,1689836716559}] 2023-07-20 07:05:22,178 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:22,178 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6707c3fc6807504cee3bc124f1adff97, NAME => 'hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:22,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:22,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,179 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,181 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:22,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b729b0d456def9e430b606c44f89497, NAME => 't,,1689836719248.0b729b0d456def9e430b606c44f89497.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:22,182 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:22,183 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:22,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689836719248.0b729b0d456def9e430b606c44f89497.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:22,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,183 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6707c3fc6807504cee3bc124f1adff97 columnFamilyName info 2023-07-20 07:05:22,184 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(310): Store=6707c3fc6807504cee3bc124f1adff97/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:22,185 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,186 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,187 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:22,187 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:22,188 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b729b0d456def9e430b606c44f89497 columnFamilyName f 2023-07-20 07:05:22,188 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,189 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(310): Store=0b729b0d456def9e430b606c44f89497/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:22,191 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000005 2023-07-20 07:05:22,192 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,194 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000013 2023-07-20 07:05:22,195 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000005 2023-07-20 07:05:22,196 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6707c3fc6807504cee3bc124f1adff97 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-20 07:05:22,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000013 2023-07-20 07:05:22,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0b729b0d456def9e430b606c44f89497 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-20 07:05:22,231 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/.tmp/info/4fe3dd8a9acd453c88c5ae82d2b0a0f6 2023-07-20 07:05:22,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/.tmp/f/b1f06193e1084185b1b9bfde90136f12 2023-07-20 07:05:22,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/.tmp/f/b1f06193e1084185b1b9bfde90136f12 as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f/b1f06193e1084185b1b9bfde90136f12 2023-07-20 07:05:22,249 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/.tmp/info/4fe3dd8a9acd453c88c5ae82d2b0a0f6 as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info/4fe3dd8a9acd453c88c5ae82d2b0a0f6 2023-07-20 07:05:22,252 WARN [RPCClient-NioEventLoopGroup-5-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33117 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:22,252 DEBUG [RPCClient-NioEventLoopGroup-5-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33117 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 2023-07-20 07:05:22,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f/b1f06193e1084185b1b9bfde90136f12, entries=10, sequenceid=13, filesize=4.8 K 2023-07-20 07:05:22,260 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for 0b729b0d456def9e430b606c44f89497 in 58ms, sequenceid=13, compaction requested=false; wal=null 2023-07-20 07:05:22,260 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-20 07:05:22,260 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info/4fe3dd8a9acd453c88c5ae82d2b0a0f6, entries=2, sequenceid=5, filesize=4.8 K 2023-07-20 07:05:22,260 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6707c3fc6807504cee3bc124f1adff97 in 65ms, sequenceid=5, compaction requested=false; wal=null 2023-07-20 07:05:22,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-20 07:05:22,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/0000000000000000013 2023-07-20 07:05:22,263 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/0000000000000000005 2023-07-20 07:05:22,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:22,302 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:22,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-20 07:05:22,308 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-20 07:05:22,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b729b0d456def9e430b606c44f89497; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68935277, jitterRate=0.02721567451953888}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:22,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:22,310 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6707c3fc6807504cee3bc124f1adff97; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66807198, jitterRate=-0.004495173692703247}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:22,310 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:22,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689836719248.0b729b0d456def9e430b606c44f89497., pid=21, masterSystemTime=1689836722172 2023-07-20 07:05:22,311 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97., pid=22, masterSystemTime=1689836722172 2023-07-20 07:05:22,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:22,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:22,314 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=19 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:22,314 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:22,314 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:22,314 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836722314"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836722314"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836722314"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836722314"}]},"ts":"1689836722314"} 2023-07-20 07:05:22,315 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:22,315 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836722315"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836722315"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836722315"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836722315"}]},"ts":"1689836722315"} 2023-07-20 07:05:22,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=19 2023-07-20 07:05:22,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,39933,1689836716559 in 297 msec 2023-07-20 07:05:22,322 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=20 2023-07-20 07:05:22,323 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=20, state=SUCCESS; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,39933,1689836716559 in 298 msec 2023-07-20 07:05:22,324 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN in 565 msec 2023-07-20 07:05:22,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-07-20 07:05:22,326 INFO [PEWorker-1] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,33117,1689836716515 after splitting done 2023-07-20 07:05:22,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN in 567 msec 2023-07-20 07:05:22,326 DEBUG [PEWorker-1] master.DeadServer(114): Removed jenkins-hbase4.apache.org,33117,1689836716515 from processing; numProcessing=0 2023-07-20 07:05:22,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,33117,1689836716515, splitWal=true, meta=true in 2.1120 sec 2023-07-20 07:05:22,463 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ef800db to 127.0.0.1:60197 2023-07-20 07:05:22,463 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:22,463 INFO [Listener at localhost/32987] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-20 07:05:22,477 INFO [Listener at localhost/32987] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=268 (was 294), OpenFileDescriptor=583 (was 581) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=128 (was 139), ProcessCount=170 (was 170), AvailableMemoryMB=5735 (was 5786) 2023-07-20 07:05:22,488 INFO [Listener at localhost/32987] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=268, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=128, ProcessCount=170, AvailableMemoryMB=5734 2023-07-20 07:05:22,489 INFO [Listener at localhost/32987] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:33117 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-20 07:05:22,501 INFO [Listener at localhost/32987] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-20 07:05:22,503 INFO [Listener at localhost/32987] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39025 2023-07-20 07:05:22,503 INFO [Listener at localhost/32987] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-20 07:05:22,504 DEBUG [Listener at localhost/32987] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-20 07:05:22,505 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:22,506 INFO [Listener at localhost/32987] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-20 07:05:22,507 INFO [Listener at localhost/32987] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39025 connecting to ZooKeeper ensemble=127.0.0.1:60197 2023-07-20 07:05:22,511 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:390250x0, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-20 07:05:22,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:39025-0x101818c7d050008 connected 2023-07-20 07:05:22,512 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-20 07:05:22,513 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-20 07:05:22,514 DEBUG [Listener at localhost/32987] zookeeper.ZKUtil(164): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-20 07:05:22,514 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39025 2023-07-20 07:05:22,515 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39025 2023-07-20 07:05:22,515 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39025 2023-07-20 07:05:22,515 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39025 2023-07-20 07:05:22,515 DEBUG [Listener at localhost/32987] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39025 2023-07-20 07:05:22,517 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(951): ClusterId : 65bacaa8-36d9-456b-9905-ec0f2285fed6 2023-07-20 07:05:22,518 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-20 07:05:22,519 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-20 07:05:22,520 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-20 07:05:22,521 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-20 07:05:22,522 DEBUG [RS:2;jenkins-hbase4:39025] zookeeper.ReadOnlyZKClient(139): Connect 0x4c9c2681 to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:22,526 DEBUG [RS:2;jenkins-hbase4:39025] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@425bc162, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:22,526 DEBUG [RS:2;jenkins-hbase4:39025] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46644df1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 07:05:22,534 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:39025 2023-07-20 07:05:22,534 INFO [RS:2;jenkins-hbase4:39025] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-20 07:05:22,534 INFO [RS:2;jenkins-hbase4:39025] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-20 07:05:22,534 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1022): About to register with Master. 2023-07-20 07:05:22,535 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase4.apache.org,40953,1689836715351 with isa=jenkins-hbase4.apache.org/172.31.14.131:39025, startcode=1689836722500 2023-07-20 07:05:22,535 DEBUG [RS:2;jenkins-hbase4:39025] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-20 07:05:22,537 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34617, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-20 07:05:22,538 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=40953] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,538 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1 2023-07-20 07:05:22,538 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost:35421 2023-07-20 07:05:22,538 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-20 07:05:22,541 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:22,541 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:22,541 DEBUG [RS:2;jenkins-hbase4:39025] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,542 WARN [RS:2;jenkins-hbase4:39025] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-20 07:05:22,542 INFO [RS:2;jenkins-hbase4:39025] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:22,542 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39025,1689836722500] 2023-07-20 07:05:22,542 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1948): logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:22,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,546 DEBUG [RS:2;jenkins-hbase4:39025] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:22,546 DEBUG [RS:2;jenkins-hbase4:39025] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,547 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-20 07:05:22,547 INFO [RS:2;jenkins-hbase4:39025] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-20 07:05:22,550 INFO [RS:2;jenkins-hbase4:39025] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-20 07:05:22,551 INFO [RS:2;jenkins-hbase4:39025] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-20 07:05:22,551 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,553 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-20 07:05:22,554 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,555 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,555 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,555 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,555 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,556 DEBUG [RS:2;jenkins-hbase4:39025] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-07-20 07:05:22,560 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,560 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,560 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,572 INFO [RS:2;jenkins-hbase4:39025] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-20 07:05:22,572 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39025,1689836722500-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-20 07:05:22,590 INFO [RS:2;jenkins-hbase4:39025] regionserver.Replication(203): jenkins-hbase4.apache.org,39025,1689836722500 started 2023-07-20 07:05:22,590 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1637): Serving as jenkins-hbase4.apache.org,39025,1689836722500, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39025, sessionid=0x101818c7d050008 2023-07-20 07:05:22,591 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase4:39025,5,FailOnTimeoutGroup] 2023-07-20 07:05:22,591 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-20 07:05:22,591 DEBUG [RS:2;jenkins-hbase4:39025] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,591 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39025,1689836722500' 2023-07-20 07:05:22,591 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-20 07:05:22,591 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:22,591 INFO [Listener at localhost/32987] client.TestScannerTimeout(177): START ************ test3686b 2023-07-20 07:05:22,591 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-20 07:05:22,592 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-20 07:05:22,592 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-20 07:05:22,592 DEBUG [RS:2;jenkins-hbase4:39025] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:22,592 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39025,1689836722500' 2023-07-20 07:05:22,592 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-20 07:05:22,592 WARN [RPCClient-NioEventLoopGroup-5-1] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:33117 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:22,593 DEBUG [RS:2;jenkins-hbase4:39025] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-20 07:05:22,594 DEBUG [RPCClient-NioEventLoopGroup-5-1] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:33117 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:33117 2023-07-20 07:05:22,595 DEBUG [RS:2;jenkins-hbase4:39025] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-20 07:05:22,595 INFO [RS:2;jenkins-hbase4:39025] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-20 07:05:22,595 INFO [RS:2;jenkins-hbase4:39025] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-20 07:05:22,697 DEBUG [hconnection-0x31add453-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:22,699 INFO [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39025%2C1689836722500, suffix=, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39025,1689836722500, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:22,700 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48452, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:22,703 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 07:05:22,703 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:22,704 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(139): Connect 0x202ed1b4 to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:22,709 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a9ecc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:22,713 DEBUG [hconnection-0x2cb714aa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:22,716 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48456, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:22,724 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:22,724 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:22,728 INFO [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39025,1689836722500/jenkins-hbase4.apache.org%2C39025%2C1689836722500.1689836722700 2023-07-20 07:05:22,728 DEBUG [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK], DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK]] 2023-07-20 07:05:23,764 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-20 07:05:25,512 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 07:05:25,513 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-20 07:05:37,727 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease -3576438783710593018 expired clientIPAndPort=172.31.14.131:48456, userName=jenkins, regionInfo=t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39933] regionserver.RSRpcServices(3094): Client tried to access missing scanner -3576438783710593018 2023-07-20 07:05:39,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39933] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 172.31.14.131:48456 deadline: 1689836799728, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-3576438783710593018'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-20 07:05:39,730 DEBUG [Listener at localhost/32987] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-3576438783710593018'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-3576438783710593018'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '-3576438783710593018'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-20 07:05:39,734 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x202ed1b4 to 127.0.0.1:60197 2023-07-20 07:05:39,734 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:39,734 INFO [Listener at localhost/32987] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-20 07:05:39,754 INFO [Listener at localhost/32987] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=297 (was 268) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1937877015_17 at /127.0.0.1:56460 [Receiving block BP-1692435750-172.31.14.131-1689836712308:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-1692435750-172.31.14.131-1689836712308:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:39025 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x4c9c2681 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/2017754218.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1-prefix:jenkins-hbase4.apache.org,39025,1689836722500 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x202ed1b4-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: jenkins-hbase4:39025Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1937877015_17 at /127.0.0.1:59410 [Receiving block BP-1692435750-172.31.14.131-1689836712308:blk_1073741847_1023] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x202ed1b4 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/2017754218.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: PacketResponder: BP-1692435750-172.31.14.131-1689836712308:blk_1073741847_1023, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x4c9c2681-SendThread(127.0.0.1:60197) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x202ed1b4-SendThread(127.0.0.1:60197) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RS:2;jenkins-hbase4:39025-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:35421 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39025 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:60197@0x4c9c2681-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=574 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 128), ProcessCount=170 (was 170), AvailableMemoryMB=5885 (was 5734) - AvailableMemoryMB LEAK? - 2023-07-20 07:05:39,788 INFO [Listener at localhost/32987] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=297, OpenFileDescriptor=574, MaxFileDescriptor=60000, SystemLoadAverage=122, ProcessCount=170, AvailableMemoryMB=5884 2023-07-20 07:05:39,789 INFO [Listener at localhost/32987] hbase.LocalHBaseCluster(262): Not alive RS:0;jenkins-hbase4:33117 2023-07-20 07:05:39,789 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase4.apache.org,33117,1689836716515 2023-07-20 07:05:39,789 INFO [Listener at localhost/32987] client.TestScannerTimeout(104): START************ test2772 2023-07-20 07:05:39,792 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-20 07:05:39,793 DEBUG [Listener at localhost/32987] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,793 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(139): Connect 0x50f46024 to 127.0.0.1:60197 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-20 07:05:39,806 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54117b10, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-20 07:05:39,807 ERROR [Listener at localhost/32987] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase4.apache.org,39933,1689836716559: die! ***** 2023-07-20 07:05:39,807 ERROR [Listener at localhost/32987] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-20 07:05:39,807 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-20 07:05:39,809 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-20 07:05:39,817 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-20 07:05:39,820 DEBUG [Listener at localhost/32987] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-20 07:05:39,892 INFO [Listener at localhost/32987] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "Verbose": false, "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1155530752, "init": 513802240, "max": 2051014656, "used": 269330168 }, "NonHeapMemoryUsage": { "committed": 124149760, "init": 2555904, "max": -1, "used": 121816976 }, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 0, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 13138, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 197, "ResponseSize_max": 679, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 199, "ResponseSize_median": 202, "ResponseSize_75th_percentile": 676, "ResponseSize_90th_percentile": 677, "ResponseSize_95th_percentile": 678, "ResponseSize_98th_percentile": 678, "ResponseSize_99th_percentile": 678, "ResponseSize_99.9th_percentile": 678, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 6, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 2, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 17196 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:60197", "tag.serverName": "jenkins-hbase4.apache.org,33117,1689836716515", "tag.clusterId": "65bacaa8-36d9-456b-9905-ec0f2285fed6", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase4.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 2, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689836716515, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 2492, "localBytesRead": 2492, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 8750, "ByteBuffAllocatorPoolAllocationBytes": 998400, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 40, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 6, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 12, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 22, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 12, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 22, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 6, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 12, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 50, "ServerReadQueryPerSecond_mean_rate": 2.2603251506525757, "ServerReadQueryPerSecond_1min_rate": 6.853446891028364, "ServerReadQueryPerSecond_5min_rate": 8.370818935606284, "ServerReadQueryPerSecond_15min_rate": 8.654548793630234, "ServerWriteQueryPerSecond_count": 26, "ServerWriteQueryPerSecond_mean_rate": 1.1753692034175554, "ServerWriteQueryPerSecond_1min_rate": 4.049764071971306, "ServerWriteQueryPerSecond_5min_rate": 4.946393007403714, "ServerWriteQueryPerSecond_15min_rate": 5.114051559872411 } ] } 2023-07-20 07:05:39,895 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=40953] master.MasterRpcServices(612): jenkins-hbase4.apache.org,39933,1689836716559 reported a fatal error: ***** ABORTING region server jenkins-hbase4.apache.org,39933,1689836716559: die! ***** 2023-07-20 07:05:39,901 INFO [Listener at localhost/32987] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39933,1689836716559' ***** 2023-07-20 07:05:39,903 INFO [Listener at localhost/32987] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-20 07:05:39,903 INFO [RS:1;jenkins-hbase4:39933] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 07:05:39,904 INFO [RS:1;jenkins-hbase4:39933] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-20 07:05:39,904 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 07:05:39,904 INFO [RS:1;jenkins-hbase4:39933] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-20 07:05:39,904 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(3305): Received CLOSE for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(3305): Received CLOSE for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1141): aborting server jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:39,905 DEBUG [RS:1;jenkins-hbase4:39933] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c60ef52 to 127.0.0.1:60197 2023-07-20 07:05:39,905 DEBUG [RS:1;jenkins-hbase4:39933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 07:05:39,905 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 07:05:39,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b729b0d456def9e430b606c44f89497, disabling compactions & flushes 2023-07-20 07:05:39,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. after waiting 0 ms 2023-07-20 07:05:39,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,914 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 07:05:39,915 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1478): Online Regions={0b729b0d456def9e430b606c44f89497=t,,1689836719248.0b729b0d456def9e430b606c44f89497., 1588230740=hbase:meta,,1.1588230740, 6707c3fc6807504cee3bc124f1adff97=hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.} 2023-07-20 07:05:39,915 DEBUG [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1504): Waiting on 0b729b0d456def9e430b606c44f89497, 1588230740, 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:39,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 07:05:39,926 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 07:05:39,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 07:05:39,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 07:05:39,926 DEBUG [hconnection-0xce8634e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:39,926 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 07:05:39,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:39,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:39,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6707c3fc6807504cee3bc124f1adff97, disabling compactions & flushes 2023-07-20 07:05:39,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:39,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:39,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. after waiting 0 ms 2023-07-20 07:05:39,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:39,947 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1825): Memstore data size is 1792 in region hbase:meta,,1.1588230740 2023-07-20 07:05:39,949 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52536, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:39,949 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 07:05:39,950 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:39,950 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 07:05:39,950 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39933] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39933,1689836716559 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 07:05:39,951 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:39,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:39,951 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39933] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:52536 deadline: 1689836799949, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39933,1689836716559 aborting 2023-07-20 07:05:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:39,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:40,058 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39933] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39933,1689836716559 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-20 07:05:40,058 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39933] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:52536 deadline: 1689836800057, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase4.apache.org,39933,1689836716559 aborting 2023-07-20 07:05:40,115 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39933,1689836716559; all regions closed. 2023-07-20 07:05:40,128 DEBUG [RS:1;jenkins-hbase4:39933] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:40,128 INFO [RS:1;jenkins-hbase4:39933] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:40,128 INFO [RS:1;jenkins-hbase4:39933] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 07:05:40,129 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 07:05:40,129 INFO [RS:1;jenkins-hbase4:39933] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39933 2023-07-20 07:05:40,135 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:40,135 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 2023-07-20 07:05:40,135 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:40,135 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:40,135 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:40,137 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39933,1689836716559] 2023-07-20 07:05:40,137 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39933,1689836716559; numProcessing=1 2023-07-20 07:05:40,138 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39933,1689836716559 already deleted, retry=false 2023-07-20 07:05:40,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,138 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase4.apache.org,39933,1689836716559 on jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:40,139 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,39933,1689836716559 znode expired, triggering replicatorRemoved event 2023-07-20 07:05:40,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,140 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=23, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase4.apache.org,39933,1689836716559, splitWal=true, meta=true 2023-07-20 07:05:40,140 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=23 for jenkins-hbase4.apache.org,39933,1689836716559 (carryingMeta=true) jenkins-hbase4.apache.org,39933,1689836716559/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@27cce485[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-20 07:05:40,142 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=23, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39933,1689836716559, splitWal=true, meta=true 2023-07-20 07:05:40,144 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39933,1689836716559, splitWal=true, meta=true, isMeta: true 2023-07-20 07:05:40,150 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting 2023-07-20 07:05:40,152 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39933,1689836716559 WAL count=1, meta=true 2023-07-20 07:05:40,152 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta}] 2023-07-20 07:05:40,155 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,157 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, worker=jenkins-hbase4.apache.org,39025,1689836722500}] 2023-07-20 07:05:40,237 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:40,237 INFO [RS:1;jenkins-hbase4:39933] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39933,1689836716559; zookeeper connection closed. 2023-07-20 07:05:40,237 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39933-0x101818c7d050002, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:40,238 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2e42c494] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2e42c494 2023-07-20 07:05:40,262 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39933 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39933 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:40,263 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39933 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39933 2023-07-20 07:05:40,310 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,311 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-20 07:05:40,312 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43280, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-20 07:05:40,313 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=39025] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=25 2023-07-20 07:05:40,334 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(300): Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, size=2.3 K (2403bytes) 2023-07-20 07:05:40,334 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta 2023-07-20 07:05:40,335 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta after 1ms 2023-07-20 07:05:40,340 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(311): Open hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta took 7ms 2023-07-20 07:05:40,344 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 13 store_sequence_id { family_name: "info" sequence_id: 15 } store_sequence_id { family_name: "rep_barrier" sequence_id: 13 } store_sequence_id { family_name: "table" sequence_id: 13 } 2023-07-20 07:05:40,345 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta so closing down 2023-07-20 07:05:40,346 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 07:05:40,353 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp 2023-07-20 07:05:40,354 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 07:05:40,369 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) 2023-07-20 07:05:40,371 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 07:05:40,372 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(423): Processed 5 edits across 1 Regions in 32 ms; skipped=1; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false 2023-07-20 07:05:40,372 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, journal: Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, size=2.3 K (2403bytes) at 1689836740334Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta so closing down at 1689836740346 (+12 ms)Creating recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp at 1689836740353 (+7 ms)3 split writer threads finished at 1689836740354 (+1 ms)Closed recovered edits writer path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp (wrote 4 edits, skipped 0 edits in 0 ms) at 1689836740369 (+15 ms)Rename recovered edits hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000016-jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta.temp to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 at 1689836740372 (+3 ms)Processed 5 edits across 1 Regions in 32 ms; skipped=1; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, size=2.3 K, length=2403, corrupted=false, cancelled=false at 1689836740372 2023-07-20 07:05:40,372 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta 2023-07-20 07:05:40,372 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=25 2023-07-20 07:05:40,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.HMaster(3641): Remote procedure done, pid=25 2023-07-20 07:05:40,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] wal.WALSplitUtil(141): Moved hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:40,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-07-20 07:05:40,380 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, worker=jenkins-hbase4.apache.org,39025,1689836722500 in 220 msec 2023-07-20 07:05:40,384 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=24, resume processing ppid=23 2023-07-20 07:05:40,387 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.meta.1689836720852.meta, worker=jenkins-hbase4.apache.org,39025,1689836722500 in 232 msec 2023-07-20 07:05:40,389 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting dir is empty, no logs to split. 2023-07-20 07:05:40,389 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39933,1689836716559 WAL count=0, meta=true 2023-07-20 07:05:40,390 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39933,1689836716559 WAL splitting is done? wals=0, meta=true 2023-07-20 07:05:40,391 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-20 07:05:40,392 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-20 07:05:40,395 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:40,534 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:40,545 DEBUG [jenkins-hbase4:40953] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 07:05:40,546 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39025,1689836722500, state=OPENING 2023-07-20 07:05:40,551 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 07:05:40,551 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39025,1689836722500}] 2023-07-20 07:05:40,552 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:40,566 WARN [ReadOnlyZKClient-127.0.0.1:60197@0x50f46024] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-20 07:05:40,567 DEBUG [hconnection-0xce8634e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:40,571 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36800, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:40,572 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39025] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 172.31.14.131:36800 deadline: 1689836800571, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,711 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-20 07:05:40,711 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-20 07:05:40,713 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39025%2C1689836722500.meta, suffix=.meta, logDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39025,1689836722500, archiveDir=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs, maxLogs=32 2023-07-20 07:05:40,735 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK] 2023-07-20 07:05:40,736 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK] 2023-07-20 07:05:40,738 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39025,1689836722500/jenkins-hbase4.apache.org%2C39025%2C1689836722500.meta.1689836740714.meta 2023-07-20 07:05:40,738 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34747,DS-1b31c1fc-e88f-4c9e-940e-cbe454a919ab,DISK], DatanodeInfoWithStorage[127.0.0.1:38425,DS-caa79fbe-0bb7-4f0e-8717-c4494d3b92cc,DISK]] 2023-07-20 07:05:40,738 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:40,738 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-20 07:05:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-20 07:05:40,739 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-20 07:05:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-20 07:05:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-20 07:05:40,739 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-20 07:05:40,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-20 07:05:40,742 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:40,743 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info 2023-07-20 07:05:40,743 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-20 07:05:40,753 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/2dab5b285a774c6c95a9da15d4b2a801 2023-07-20 07:05:40,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:40,754 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-20 07:05:40,757 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:40,757 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/rep_barrier 2023-07-20 07:05:40,758 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-20 07:05:40,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:40,759 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-20 07:05:40,760 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:40,760 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table 2023-07-20 07:05:40,760 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-20 07:05:40,771 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(539): loaded hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/table/e33f477895ff430fa59b695507099b94 2023-07-20 07:05:40,771 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:40,773 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:40,775 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740 2023-07-20 07:05:40,776 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 07:05:40,781 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5464): Applied 16, skipped 0, firstSequenceIdInLog=16, maxSequenceIdInLog=19, path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 07:05:40,781 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 07:05:40,803 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=19 (bloomFilter=false), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/c4f8f5e9d52e4130b2a38e49143a1360 2023-07-20 07:05:40,813 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/c4f8f5e9d52e4130b2a38e49143a1360 as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/c4f8f5e9d52e4130b2a38e49143a1360 2023-07-20 07:05:40,824 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/c4f8f5e9d52e4130b2a38e49143a1360, entries=16, sequenceid=19, filesize=6.4 K 2023-07-20 07:05:40,825 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 44ms, sequenceid=19, compaction requested=false; wal=null 2023-07-20 07:05:40,826 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/0000000000000000019 2023-07-20 07:05:40,832 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-20 07:05:40,834 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-20 07:05:40,838 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=13 2023-07-20 07:05:40,840 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=20; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64624346, jitterRate=-0.03702220320701599}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-20 07:05:40,840 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-20 07:05:40,841 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=27, masterSystemTime=1689836740705 2023-07-20 07:05:40,844 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-20 07:05:40,844 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-20 07:05:40,845 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39025,1689836722500, state=OPEN 2023-07-20 07:05:40,846 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-20 07:05:40,846 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-20 07:05:40,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-07-20 07:05:40,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39025,1689836722500 in 295 msec 2023-07-20 07:05:40,851 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=23 2023-07-20 07:05:40,851 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase4.apache.org,39933,1689836716559 had 3 regions 2023-07-20 07:05:40,852 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 458 msec 2023-07-20 07:05:40,852 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=23, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase4.apache.org,39933,1689836716559, splitWal=true, meta=true, isMeta: false 2023-07-20 07:05:40,855 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39933,1689836716559 WAL count=1, meta=false 2023-07-20 07:05:40,855 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=28, ppid=23, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958}] 2023-07-20 07:05:40,857 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:40,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, worker=jenkins-hbase4.apache.org,39025,1689836722500}] 2023-07-20 07:05:41,011 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=39025] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=29 2023-07-20 07:05:41,024 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(300): Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, size=721 (721bytes) 2023-07-20 07:05:41,024 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 2023-07-20 07:05:41,025 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 after 1ms 2023-07-20 07:05:41,028 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(311): Open hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 took 4ms 2023-07-20 07:05:41,032 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 0b729b0d456def9e430b606c44f89497: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-20 07:05:41,033 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(346): Last flushed sequenceid for 6707c3fc6807504cee3bc124f1adff97: last_flushed_sequence_id: 5 store_sequence_id { family_name: "info" sequence_id: 5 } 2023-07-20 07:05:41,033 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 so closing down 2023-07-20 07:05:41,033 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-20 07:05:41,034 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-20 07:05:41,034 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(423): Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, size=721, length=721, corrupted=false, cancelled=false 2023-07-20 07:05:41,034 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, journal: Splitting hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, size=721 (721bytes) at 1689836741024Finishing writing output for hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 so closing down at 1689836741033 (+9 ms)3 split writer threads finished at 1689836741034 (+1 ms)Processed 2 edits across 0 Regions in 5 ms; skipped=2; WAL=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, size=721, length=721, corrupted=false, cancelled=false at 1689836741034 2023-07-20 07:05:41,034 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 2023-07-20 07:05:41,034 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=29 2023-07-20 07:05:41,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] master.HMaster(3641): Remote procedure done, pid=29 2023-07-20 07:05:41,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40953] wal.WALSplitUtil(141): Moved hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting/jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958 to hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:41,040 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-07-20 07:05:41,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, worker=jenkins-hbase4.apache.org,39025,1689836722500 in 180 msec 2023-07-20 07:05:41,042 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,044 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=28, resume processing ppid=23 2023-07-20 07:05:41,044 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=23, state=SUCCESS; SplitWALProcedure jenkins-hbase4.apache.org%2C39933%2C1689836716559.1689836717958, worker=jenkins-hbase4.apache.org,39025,1689836722500 in 187 msec 2023-07-20 07:05:41,046 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/WALs/jenkins-hbase4.apache.org,39933,1689836716559-splitting dir is empty, no logs to split. 2023-07-20 07:05:41,046 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase4.apache.org,39933,1689836716559 WAL count=0, meta=false 2023-07-20 07:05:41,046 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase4.apache.org,39933,1689836716559 WAL splitting is done? wals=0, meta=false 2023-07-20 07:05:41,048 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN}, {pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN}] 2023-07-20 07:05:41,050 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN 2023-07-20 07:05:41,050 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN 2023-07-20 07:05:41,051 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:41,051 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=30, ppid=23, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-20 07:05:41,080 DEBUG [hconnection-0xce8634e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39933 this server is in the failed servers list 2023-07-20 07:05:41,188 DEBUG [hconnection-0xce8634e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39933 this server is in the failed servers list 2023-07-20 07:05:41,201 DEBUG [jenkins-hbase4:40953] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-20 07:05:41,201 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,201 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,202 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836741201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836741201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836741201"}]},"ts":"1689836741201"} 2023-07-20 07:05:41,202 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836741201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1689836741201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689836741201"}]},"ts":"1689836741201"} 2023-07-20 07:05:41,202 WARN [RS-EventLoopGroup-4-3] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase4.apache.org/172.31.14.131:39933 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39933 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-20 07:05:41,203 DEBUG [RS-EventLoopGroup-4-3] ipc.FailedServers(52): Added failed server with address jenkins-hbase4.apache.org/172.31.14.131:39933 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase4.apache.org/172.31.14.131:39933 2023-07-20 07:05:41,304 DEBUG [PEWorker-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-20 07:05:41,306 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36814, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-20 07:05:41,308 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=30, state=RUNNABLE; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,39025,1689836722500}] 2023-07-20 07:05:41,309 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,39025,1689836722500}] 2023-07-20 07:05:41,393 DEBUG [hconnection-0xce8634e-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase4.apache.org/172.31.14.131:39933 this server is in the failed servers list 2023-07-20 07:05:41,465 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,465 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6707c3fc6807504cee3bc124f1adff97, NAME => 'hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:41,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:41,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,466 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,468 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(130): Open t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0b729b0d456def9e430b606c44f89497, NAME => 't,,1689836719248.0b729b0d456def9e430b606c44f89497.', STARTKEY => '', ENDKEY => ''} 2023-07-20 07:05:41,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,469 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:41,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated t,,1689836719248.0b729b0d456def9e430b606c44f89497.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-20 07:05:41,469 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info 2023-07-20 07:05:41,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,470 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6707c3fc6807504cee3bc124f1adff97 columnFamilyName info 2023-07-20 07:05:41,472 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,473 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:41,473 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f 2023-07-20 07:05:41,474 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0b729b0d456def9e430b606c44f89497 columnFamilyName f 2023-07-20 07:05:41,479 DEBUG [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(539): loaded hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/info/4fe3dd8a9acd453c88c5ae82d2b0a0f6 2023-07-20 07:05:41,480 INFO [StoreOpener-6707c3fc6807504cee3bc124f1adff97-1] regionserver.HStore(310): Store=6707c3fc6807504cee3bc124f1adff97/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:41,481 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,483 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,484 DEBUG [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(539): loaded hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/f/b1f06193e1084185b1b9bfde90136f12 2023-07-20 07:05:41,484 INFO [StoreOpener-0b729b0d456def9e430b606c44f89497-1] regionserver.HStore(310): Store=0b729b0d456def9e430b606c44f89497/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-20 07:05:41,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,487 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,488 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,491 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/6.seqid, newMaxSeqId=6, maxSeqId=5 2023-07-20 07:05:41,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,493 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6707c3fc6807504cee3bc124f1adff97; next sequenceid=7; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75378538, jitterRate=0.12322774529457092}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:41,493 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:41,494 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97., pid=33, masterSystemTime=1689836741460 2023-07-20 07:05:41,494 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-20 07:05:41,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0b729b0d456def9e430b606c44f89497; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73032785, jitterRate=0.0882733017206192}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-20 07:05:41,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:41,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689836719248.0b729b0d456def9e430b606c44f89497., pid=32, masterSystemTime=1689836741460 2023-07-20 07:05:41,497 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=6707c3fc6807504cee3bc124f1adff97, regionState=OPEN, openSeqNum=7, regionLocation=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,497 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689836741497"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836741497"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836741497"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836741497"}]},"ts":"1689836741497"} 2023-07-20 07:05:41,499 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,499 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,500 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,500 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=30 updating hbase:meta row=0b729b0d456def9e430b606c44f89497, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,501 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689836719248.0b729b0d456def9e430b606c44f89497.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689836741500"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1689836741500"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689836741500"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689836741500"}]},"ts":"1689836741500"} 2023-07-20 07:05:41,503 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-07-20 07:05:41,503 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 6707c3fc6807504cee3bc124f1adff97, server=jenkins-hbase4.apache.org,39025,1689836722500 in 190 msec 2023-07-20 07:05:41,505 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6707c3fc6807504cee3bc124f1adff97, ASSIGN in 455 msec 2023-07-20 07:05:41,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=30 2023-07-20 07:05:41,505 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=30, state=SUCCESS; OpenRegionProcedure 0b729b0d456def9e430b606c44f89497, server=jenkins-hbase4.apache.org,39025,1689836722500 in 195 msec 2023-07-20 07:05:41,507 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-07-20 07:05:41,507 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase4.apache.org,39933,1689836716559 after splitting done 2023-07-20 07:05:41,508 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; TransitRegionStateProcedure table=t, region=0b729b0d456def9e430b606c44f89497, ASSIGN in 457 msec 2023-07-20 07:05:41,508 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase4.apache.org,39933,1689836716559 from processing; numProcessing=0 2023-07-20 07:05:41,509 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=23, state=SUCCESS; ServerCrashProcedure jenkins-hbase4.apache.org,39933,1689836716559, splitWal=true, meta=true in 1.3690 sec 2023-07-20 07:05:41,705 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50f46024 to 127.0.0.1:60197 2023-07-20 07:05:41,705 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,705 INFO [Listener at localhost/32987] client.TestScannerTimeout(124): END ************ test2772 2023-07-20 07:05:41,721 INFO [Listener at localhost/32987] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=278 (was 297), OpenFileDescriptor=577 (was 574) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=137 (was 122) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=5899 (was 5884) - AvailableMemoryMB LEAK? - 2023-07-20 07:05:41,722 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-20 07:05:41,722 INFO [Listener at localhost/32987] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-20 07:05:41,722 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5363c3cc to 127.0.0.1:60197 2023-07-20 07:05:41,723 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,723 DEBUG [Listener at localhost/32987] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-20 07:05:41,723 DEBUG [Listener at localhost/32987] util.JVMClusterUtil(257): Found active master hash=233508700, stopped=false 2023-07-20 07:05:41,723 INFO [Listener at localhost/32987] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:41,726 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 07:05:41,726 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-20 07:05:41,726 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:41,726 INFO [Listener at localhost/32987] procedure2.ProcedureExecutor(629): Stopping 2023-07-20 07:05:41,726 DEBUG [Listener at localhost/32987] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2689565b to 127.0.0.1:60197 2023-07-20 07:05:41,727 DEBUG [Listener at localhost/32987] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,727 INFO [Listener at localhost/32987] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,39025,1689836722500' ***** 2023-07-20 07:05:41,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 07:05:41,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-20 07:05:41,727 INFO [Listener at localhost/32987] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-20 07:05:41,727 INFO [RS:2;jenkins-hbase4:39025] regionserver.HeapMemoryManager(220): Stopping 2023-07-20 07:05:41,727 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-20 07:05:41,727 INFO [RS:2;jenkins-hbase4:39025] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-20 07:05:41,727 INFO [RS:2;jenkins-hbase4:39025] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-20 07:05:41,728 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(3305): Received CLOSE for 0b729b0d456def9e430b606c44f89497 2023-07-20 07:05:41,728 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(3305): Received CLOSE for 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,728 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,728 DEBUG [RS:2;jenkins-hbase4:39025] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c9c2681 to 127.0.0.1:60197 2023-07-20 07:05:41,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0b729b0d456def9e430b606c44f89497, disabling compactions & flushes 2023-07-20 07:05:41,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,728 DEBUG [RS:2;jenkins-hbase4:39025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,728 INFO [RS:2;jenkins-hbase4:39025] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-20 07:05:41,729 INFO [RS:2;jenkins-hbase4:39025] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-20 07:05:41,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689836719248.0b729b0d456def9e430b606c44f89497. after waiting 0 ms 2023-07-20 07:05:41,729 INFO [RS:2;jenkins-hbase4:39025] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-20 07:05:41,729 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,729 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-20 07:05:41,729 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-20 07:05:41,729 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1478): Online Regions={0b729b0d456def9e430b606c44f89497=t,,1689836719248.0b729b0d456def9e430b606c44f89497., 1588230740=hbase:meta,,1.1588230740, 6707c3fc6807504cee3bc124f1adff97=hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97.} 2023-07-20 07:05:41,729 DEBUG [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1504): Waiting on 0b729b0d456def9e430b606c44f89497, 1588230740, 6707c3fc6807504cee3bc124f1adff97 2023-07-20 07:05:41,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-20 07:05:41,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-20 07:05:41,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-20 07:05:41,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-20 07:05:41,730 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-20 07:05:41,730 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-20 07:05:41,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/default/t/0b729b0d456def9e430b606c44f89497/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-20 07:05:41,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0b729b0d456def9e430b606c44f89497: 2023-07-20 07:05:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed t,,1689836719248.0b729b0d456def9e430b606c44f89497. 2023-07-20 07:05:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6707c3fc6807504cee3bc124f1adff97, disabling compactions & flushes 2023-07-20 07:05:41,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. after waiting 0 ms 2023-07-20 07:05:41,743 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/namespace/6707c3fc6807504cee3bc124f1adff97/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=6 2023-07-20 07:05:41,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6707c3fc6807504cee3bc124f1adff97: 2023-07-20 07:05:41,751 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689836718449.6707c3fc6807504cee3bc124f1adff97. 2023-07-20 07:05:41,764 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=26 (bloomFilter=false), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/9a664af73c864c2d9a1c7c1b00449ddc 2023-07-20 07:05:41,772 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/.tmp/info/9a664af73c864c2d9a1c7c1b00449ddc as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/9a664af73c864c2d9a1c7c1b00449ddc 2023-07-20 07:05:41,779 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/info/9a664af73c864c2d9a1c7c1b00449ddc, entries=16, sequenceid=26, filesize=6.4 K 2023-07-20 07:05:41,780 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1792, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 50ms, sequenceid=26, compaction requested=true 2023-07-20 07:05:41,789 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/data/hbase/meta/1588230740/recovered.edits/29.seqid, newMaxSeqId=29, maxSeqId=19 2023-07-20 07:05:41,790 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-20 07:05:41,790 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:41,790 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-20 07:05:41,790 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-20 07:05:41,930 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,39025,1689836722500; all regions closed. 2023-07-20 07:05:41,936 DEBUG [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:41,936 INFO [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39025%2C1689836722500.meta:.meta(num 1689836740714) 2023-07-20 07:05:41,942 DEBUG [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/oldWALs 2023-07-20 07:05:41,942 INFO [RS:2;jenkins-hbase4:39025] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39025%2C1689836722500:(num 1689836722700) 2023-07-20 07:05:41,942 DEBUG [RS:2;jenkins-hbase4:39025] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,942 INFO [RS:2;jenkins-hbase4:39025] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:41,942 INFO [RS:2;jenkins-hbase4:39025] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-20 07:05:41,942 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 07:05:41,943 INFO [RS:2;jenkins-hbase4:39025] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39025 2023-07-20 07:05:41,946 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39025,1689836722500 2023-07-20 07:05:41,946 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:41,946 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-20 07:05:41,948 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39025,1689836722500] 2023-07-20 07:05:41,948 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39025,1689836722500; numProcessing=1 2023-07-20 07:05:41,949 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39025,1689836722500 already deleted, retry=false 2023-07-20 07:05:41,949 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39025,1689836722500 expired; onlineServers=0 2023-07-20 07:05:41,949 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase4.apache.org,40953,1689836715351' ***** 2023-07-20 07:05:41,949 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-20 07:05:41,950 DEBUG [M:0;jenkins-hbase4:40953] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cd60567, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-07-20 07:05:41,950 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegionServer(1144): stopping server jenkins-hbase4.apache.org,40953,1689836715351 2023-07-20 07:05:41,950 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegionServer(1170): stopping server jenkins-hbase4.apache.org,40953,1689836715351; all regions closed. 2023-07-20 07:05:41,950 DEBUG [M:0;jenkins-hbase4:40953] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-20 07:05:41,950 DEBUG [M:0;jenkins-hbase4:40953] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-20 07:05:41,950 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-20 07:05:41,950 DEBUG [M:0;jenkins-hbase4:40953] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-20 07:05:41,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689836717569] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1689836717569,5,FailOnTimeoutGroup] 2023-07-20 07:05:41,950 INFO [M:0;jenkins-hbase4:40953] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-20 07:05:41,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689836717570] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1689836717570,5,FailOnTimeoutGroup] 2023-07-20 07:05:41,951 INFO [M:0;jenkins-hbase4:40953] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-20 07:05:41,951 INFO [M:0;jenkins-hbase4:40953] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-07-20 07:05:41,951 DEBUG [M:0;jenkins-hbase4:40953] master.HMaster(1512): Stopping service threads 2023-07-20 07:05:41,951 INFO [M:0;jenkins-hbase4:40953] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-20 07:05:41,951 ERROR [M:0;jenkins-hbase4:40953] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (1743606826) connection to localhost/127.0.0.1:35421 from jenkins,5,PEWorkerGroup] 2023-07-20 07:05:41,952 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-20 07:05:41,952 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-20 07:05:41,952 INFO [M:0;jenkins-hbase4:40953] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-20 07:05:41,952 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-20 07:05:41,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-20 07:05:41,953 DEBUG [M:0;jenkins-hbase4:40953] zookeeper.ZKUtil(398): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-20 07:05:41,953 WARN [M:0;jenkins-hbase4:40953] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-20 07:05:41,953 INFO [M:0;jenkins-hbase4:40953] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-20 07:05:41,953 INFO [M:0;jenkins-hbase4:40953] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-20 07:05:41,954 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-20 07:05:41,954 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:41,954 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:41,954 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-20 07:05:41,954 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:41,954 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=111.32 KB heapSize=134.27 KB 2023-07-20 07:05:41,971 INFO [M:0;jenkins-hbase4:40953] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=111.32 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/dda685fc47384dde97dd91fbeb8c602e 2023-07-20 07:05:41,979 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/dda685fc47384dde97dd91fbeb8c602e as hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/dda685fc47384dde97dd91fbeb8c602e 2023-07-20 07:05:41,984 INFO [M:0;jenkins-hbase4:40953] regionserver.HStore(1080): Added hdfs://localhost:35421/user/jenkins/test-data/fca271d4-45e0-658b-d14e-8fa2daeb00f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/dda685fc47384dde97dd91fbeb8c602e, entries=33, sequenceid=257, filesize=8.5 K 2023-07-20 07:05:41,986 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegion(2948): Finished flush of dataSize ~111.32 KB/113995, heapSize ~134.26 KB/137480, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=257, compaction requested=false 2023-07-20 07:05:41,987 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-20 07:05:41,987 DEBUG [M:0;jenkins-hbase4:40953] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-20 07:05:41,991 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-20 07:05:41,991 INFO [M:0;jenkins-hbase4:40953] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-20 07:05:41,991 INFO [M:0;jenkins-hbase4:40953] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40953 2023-07-20 07:05:41,994 DEBUG [M:0;jenkins-hbase4:40953] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40953,1689836715351 already deleted, retry=false 2023-07-20 07:05:42,209 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:42,209 INFO [M:0;jenkins-hbase4:40953] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,40953,1689836715351; zookeeper connection closed. 2023-07-20 07:05:42,209 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): master:40953-0x101818c7d050000, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:42,309 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:42,309 INFO [RS:2;jenkins-hbase4:39025] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase4.apache.org,39025,1689836722500; zookeeper connection closed. 2023-07-20 07:05:42,309 DEBUG [Listener at localhost/32987-EventThread] zookeeper.ZKWatcher(600): regionserver:39025-0x101818c7d050008, quorum=127.0.0.1:60197, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-20 07:05:42,310 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@40272b9b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@40272b9b 2023-07-20 07:05:42,310 INFO [Listener at localhost/32987] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-20 07:05:42,311 WARN [Listener at localhost/32987] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 07:05:42,314 INFO [Listener at localhost/32987] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 07:05:42,423 WARN [BP-1692435750-172.31.14.131-1689836712308 heartbeating to localhost/127.0.0.1:35421] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 07:05:42,423 WARN [BP-1692435750-172.31.14.131-1689836712308 heartbeating to localhost/127.0.0.1:35421] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1692435750-172.31.14.131-1689836712308 (Datanode Uuid 7cc6d6db-1fef-40cb-a5b5-fab91404c01b) service to localhost/127.0.0.1:35421 2023-07-20 07:05:42,425 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data3/current/BP-1692435750-172.31.14.131-1689836712308] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 07:05:42,425 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data4/current/BP-1692435750-172.31.14.131-1689836712308] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 07:05:42,427 WARN [Listener at localhost/32987] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-20 07:05:42,435 INFO [Listener at localhost/32987] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 07:05:42,443 WARN [BP-1692435750-172.31.14.131-1689836712308 heartbeating to localhost/127.0.0.1:35421] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-20 07:05:42,443 WARN [BP-1692435750-172.31.14.131-1689836712308 heartbeating to localhost/127.0.0.1:35421] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1692435750-172.31.14.131-1689836712308 (Datanode Uuid d1e4ee74-d84d-48d9-9600-b82ac8fc5a3b) service to localhost/127.0.0.1:35421 2023-07-20 07:05:42,444 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data1/current/BP-1692435750-172.31.14.131-1689836712308] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 07:05:42,445 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ce5a1c26-6de4-a1a0-0a6e-354dab34e249/cluster_d8d2438f-5858-2b7f-01fc-e7ffc6d348ab/dfs/data/data2/current/BP-1692435750-172.31.14.131-1689836712308] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-20 07:05:42,482 INFO [Listener at localhost/32987] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-20 07:05:42,562 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-20 07:05:42,595 INFO [Listener at localhost/32987] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-20 07:05:42,636 INFO [Listener at localhost/32987] hbase.HBaseTestingUtility(1293): Minicluster is down