2023-07-14 19:03:03,794 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestScannerTimeout timeout: 13 mins 2023-07-14 19:03:04,014 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc 2023-07-14 19:03:04,030 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=2, rsPorts=, rsClass=null, numDataNodes=2, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-07-14 19:03:04,030 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7, deleteOnExit=true 2023-07-14 19:03:04,030 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-07-14 19:03:04,031 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/test.cache.data in system properties and HBase conf 2023-07-14 19:03:04,031 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/hadoop.tmp.dir in system properties and HBase conf 2023-07-14 19:03:04,032 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/hadoop.log.dir in system properties and HBase conf 2023-07-14 19:03:04,032 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-07-14 19:03:04,032 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-07-14 19:03:04,033 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-07-14 19:03:04,133 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-07-14 19:03:04,443 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-07-14 19:03:04,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-07-14 19:03:04,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-07-14 19:03:04,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-07-14 19:03:04,447 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 19:03:04,448 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-07-14 19:03:04,448 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-07-14 19:03:04,448 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-07-14 19:03:04,449 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 19:03:04,449 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-07-14 19:03:04,449 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/nfs.dump.dir in system properties and HBase conf 2023-07-14 19:03:04,450 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/java.io.tmpdir in system properties and HBase conf 2023-07-14 19:03:04,450 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-07-14 19:03:04,450 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-07-14 19:03:04,451 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-07-14 19:03:04,846 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 19:03:04,849 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 19:03:05,070 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-07-14 19:03:05,208 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-07-14 19:03:05,221 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 19:03:05,250 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 19:03:05,276 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/java.io.tmpdir/Jetty_localhost_localdomain_33459_hdfs____8eznma/webapp 2023-07-14 19:03:05,403 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:33459 2023-07-14 19:03:05,410 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-07-14 19:03:05,410 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-07-14 19:03:05,724 WARN [Listener at localhost.localdomain/40151] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 19:03:05,822 WARN [Listener at localhost.localdomain/40151] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 19:03:05,836 WARN [Listener at localhost.localdomain/40151] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 19:03:05,842 INFO [Listener at localhost.localdomain/40151] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 19:03:05,845 INFO [Listener at localhost.localdomain/40151] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/java.io.tmpdir/Jetty_localhost_34557_datanode____5vh75p/webapp 2023-07-14 19:03:05,923 INFO [Listener at localhost.localdomain/40151] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34557 2023-07-14 19:03:06,185 WARN [Listener at localhost.localdomain/42225] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 19:03:06,198 WARN [Listener at localhost.localdomain/42225] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-07-14 19:03:06,201 WARN [Listener at localhost.localdomain/42225] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-07-14 19:03:06,204 INFO [Listener at localhost.localdomain/42225] log.Slf4jLog(67): jetty-6.1.26 2023-07-14 19:03:06,210 INFO [Listener at localhost.localdomain/42225] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/java.io.tmpdir/Jetty_localhost_43667_datanode____.hre0sh/webapp 2023-07-14 19:03:06,327 INFO [Listener at localhost.localdomain/42225] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43667 2023-07-14 19:03:06,338 WARN [Listener at localhost.localdomain/40851] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-07-14 19:03:06,577 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1692a36fcb6b3bd: Processing first storage report for DS-d3937e71-9e9e-4434-9e6c-a1eac471555a from datanode e84c29ef-a447-41f7-afb3-6846372b9cea 2023-07-14 19:03:06,579 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1692a36fcb6b3bd: from storage DS-d3937e71-9e9e-4434-9e6c-a1eac471555a node DatanodeRegistration(127.0.0.1:42661, datanodeUuid=e84c29ef-a447-41f7-afb3-6846372b9cea, infoPort=35001, infoSecurePort=0, ipcPort=42225, storageInfo=lv=-57;cid=testClusterID;nsid=627899296;c=1689361384905), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-07-14 19:03:06,579 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa78bad14508cff77: Processing first storage report for DS-cf27f661-e167-4e69-8b53-266bf6d793d8 from datanode 59942dd0-ebfc-489d-ae82-eba6e968d81c 2023-07-14 19:03:06,579 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa78bad14508cff77: from storage DS-cf27f661-e167-4e69-8b53-266bf6d793d8 node DatanodeRegistration(127.0.0.1:36773, datanodeUuid=59942dd0-ebfc-489d-ae82-eba6e968d81c, infoPort=46677, infoSecurePort=0, ipcPort=40851, storageInfo=lv=-57;cid=testClusterID;nsid=627899296;c=1689361384905), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 19:03:06,579 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1692a36fcb6b3bd: Processing first storage report for DS-2540f9a8-61d9-4a15-90a8-fb5243cc7623 from datanode e84c29ef-a447-41f7-afb3-6846372b9cea 2023-07-14 19:03:06,579 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1692a36fcb6b3bd: from storage DS-2540f9a8-61d9-4a15-90a8-fb5243cc7623 node DatanodeRegistration(127.0.0.1:42661, datanodeUuid=e84c29ef-a447-41f7-afb3-6846372b9cea, infoPort=35001, infoSecurePort=0, ipcPort=42225, storageInfo=lv=-57;cid=testClusterID;nsid=627899296;c=1689361384905), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 19:03:06,580 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa78bad14508cff77: Processing first storage report for DS-6fd040e7-4299-43ff-b17a-7cce53070c49 from datanode 59942dd0-ebfc-489d-ae82-eba6e968d81c 2023-07-14 19:03:06,580 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa78bad14508cff77: from storage DS-6fd040e7-4299-43ff-b17a-7cce53070c49 node DatanodeRegistration(127.0.0.1:36773, datanodeUuid=59942dd0-ebfc-489d-ae82-eba6e968d81c, infoPort=46677, infoSecurePort=0, ipcPort=40851, storageInfo=lv=-57;cid=testClusterID;nsid=627899296;c=1689361384905), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-07-14 19:03:06,716 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc 2023-07-14 19:03:06,768 INFO [Listener at localhost.localdomain/40851] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/zookeeper_0, clientPort=51669, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-07-14 19:03:06,781 INFO [Listener at localhost.localdomain/40851] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=51669 2023-07-14 19:03:06,788 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:06,790 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:07,400 INFO [Listener at localhost.localdomain/40851] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b with version=8 2023-07-14 19:03:07,400 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/hbase-staging 2023-07-14 19:03:07,650 INFO [Listener at localhost.localdomain/40851] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-07-14 19:03:08,015 INFO [Listener at localhost.localdomain/40851] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-07-14 19:03:08,040 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,041 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,041 INFO [Listener at localhost.localdomain/40851] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 19:03:08,042 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,042 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 19:03:08,167 INFO [Listener at localhost.localdomain/40851] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 19:03:08,223 DEBUG [Listener at localhost.localdomain/40851] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-07-14 19:03:08,300 INFO [Listener at localhost.localdomain/40851] ipc.NettyRpcServer(120): Bind to /148.251.75.209:38349 2023-07-14 19:03:08,308 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,310 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,327 INFO [Listener at localhost.localdomain/40851] zookeeper.RecoverableZooKeeper(93): Process identifier=master:38349 connecting to ZooKeeper ensemble=127.0.0.1:51669 2023-07-14 19:03:08,359 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:383490x0, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 19:03:08,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): master:38349-0x1008cdd89820000 connected 2023-07-14 19:03:08,396 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 19:03:08,398 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 19:03:08,403 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 19:03:08,412 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38349 2023-07-14 19:03:08,414 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38349 2023-07-14 19:03:08,414 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38349 2023-07-14 19:03:08,415 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38349 2023-07-14 19:03:08,415 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38349 2023-07-14 19:03:08,420 INFO [Listener at localhost.localdomain/40851] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b, hbase.cluster.distributed=false 2023-07-14 19:03:08,479 INFO [Listener at localhost.localdomain/40851] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-14 19:03:08,480 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,480 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,480 INFO [Listener at localhost.localdomain/40851] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 19:03:08,480 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,480 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 19:03:08,484 INFO [Listener at localhost.localdomain/40851] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 19:03:08,487 INFO [Listener at localhost.localdomain/40851] ipc.NettyRpcServer(120): Bind to /148.251.75.209:34333 2023-07-14 19:03:08,489 INFO [Listener at localhost.localdomain/40851] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 19:03:08,493 DEBUG [Listener at localhost.localdomain/40851] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 19:03:08,495 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,496 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,498 INFO [Listener at localhost.localdomain/40851] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34333 connecting to ZooKeeper ensemble=127.0.0.1:51669 2023-07-14 19:03:08,502 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:343330x0, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 19:03:08,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:34333-0x1008cdd89820001 connected 2023-07-14 19:03:08,503 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 19:03:08,504 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 19:03:08,505 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 19:03:08,506 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34333 2023-07-14 19:03:08,506 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34333 2023-07-14 19:03:08,507 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34333 2023-07-14 19:03:08,508 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34333 2023-07-14 19:03:08,508 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34333 2023-07-14 19:03:08,521 INFO [Listener at localhost.localdomain/40851] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-14 19:03:08,521 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,521 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,521 INFO [Listener at localhost.localdomain/40851] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 19:03:08,522 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:08,522 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 19:03:08,522 INFO [Listener at localhost.localdomain/40851] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 19:03:08,524 INFO [Listener at localhost.localdomain/40851] ipc.NettyRpcServer(120): Bind to /148.251.75.209:42249 2023-07-14 19:03:08,525 INFO [Listener at localhost.localdomain/40851] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 19:03:08,526 DEBUG [Listener at localhost.localdomain/40851] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 19:03:08,527 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,529 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,531 INFO [Listener at localhost.localdomain/40851] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42249 connecting to ZooKeeper ensemble=127.0.0.1:51669 2023-07-14 19:03:08,535 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:422490x0, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 19:03:08,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:42249-0x1008cdd89820002 connected 2023-07-14 19:03:08,537 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-07-14 19:03:08,538 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 19:03:08,538 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 19:03:08,539 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42249 2023-07-14 19:03:08,539 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42249 2023-07-14 19:03:08,540 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42249 2023-07-14 19:03:08,540 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42249 2023-07-14 19:03:08,541 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42249 2023-07-14 19:03:08,542 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:08,550 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 19:03:08,551 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:08,572 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 19:03:08,572 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 19:03:08,572 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-07-14 19:03:08,572 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:08,573 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 19:03:08,574 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,38349,1689361387512 from backup master directory 2023-07-14 19:03:08,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 19:03:08,577 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:08,577 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-07-14 19:03:08,578 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 19:03:08,578 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:08,580 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-07-14 19:03:08,582 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-07-14 19:03:08,665 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/hbase.id with ID: cfbfac95-a50f-45d8-a4bb-94c79ffe7eeb 2023-07-14 19:03:08,717 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:08,731 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:08,776 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x7d0880e4 to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:08,803 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2272fa75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:08,828 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 19:03:08,830 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-07-14 19:03:08,850 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-07-14 19:03:08,850 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-07-14 19:03:08,852 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:08,856 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:08,857 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:08,885 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store-tmp 2023-07-14 19:03:08,916 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:08,916 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 19:03:08,917 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:08,917 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:08,917 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 19:03:08,917 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:08,917 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:08,917 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 19:03:08,919 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/WALs/jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:08,938 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38349%2C1689361387512, suffix=, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/WALs/jenkins-hbase20.apache.org,38349,1689361387512, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/oldWALs, maxLogs=10 2023-07-14 19:03:08,987 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:08,987 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:08,994 DEBUG [RS-EventLoopGroup-4-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:09,049 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/WALs/jenkins-hbase20.apache.org,38349,1689361387512/jenkins-hbase20.apache.org%2C38349%2C1689361387512.1689361388946 2023-07-14 19:03:09,050 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK], DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK]] 2023-07-14 19:03:09,050 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:09,051 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:09,054 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,054 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,103 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,110 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-07-14 19:03:09,130 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-07-14 19:03:09,140 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:09,147 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,148 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,162 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-07-14 19:03:09,167 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 19:03:09,168 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61687220, jitterRate=-0.08078879117965698}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:09,168 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 19:03:09,169 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-07-14 19:03:09,185 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-07-14 19:03:09,185 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-07-14 19:03:09,187 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-07-14 19:03:09,188 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-07-14 19:03:09,213 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 24 msec 2023-07-14 19:03:09,213 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(96): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-07-14 19:03:09,234 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-07-14 19:03:09,241 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-07-14 19:03:09,265 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-07-14 19:03:09,268 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-07-14 19:03:09,269 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-07-14 19:03:09,273 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-07-14 19:03:09,277 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-07-14 19:03:09,279 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:09,281 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-07-14 19:03:09,281 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-07-14 19:03:09,292 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-07-14 19:03:09,295 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 19:03:09,295 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 19:03:09,295 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-07-14 19:03:09,296 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:09,296 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,38349,1689361387512, sessionid=0x1008cdd89820000, setting cluster-up flag (Was=false) 2023-07-14 19:03:09,308 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:09,312 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-07-14 19:03:09,314 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:09,317 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:09,321 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-07-14 19:03:09,322 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:09,324 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.hbase-snapshot/.tmp 2023-07-14 19:03:09,344 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(951): ClusterId : cfbfac95-a50f-45d8-a4bb-94c79ffe7eeb 2023-07-14 19:03:09,344 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(951): ClusterId : cfbfac95-a50f-45d8-a4bb-94c79ffe7eeb 2023-07-14 19:03:09,349 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 19:03:09,349 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 19:03:09,354 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 19:03:09,354 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 19:03:09,354 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 19:03:09,354 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 19:03:09,357 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 19:03:09,357 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 19:03:09,359 DEBUG [RS:1;jenkins-hbase20:42249] zookeeper.ReadOnlyZKClient(139): Connect 0x1ef0dd52 to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:09,359 DEBUG [RS:0;jenkins-hbase20:34333] zookeeper.ReadOnlyZKClient(139): Connect 0x1569481a to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:09,372 DEBUG [RS:0;jenkins-hbase20:34333] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1baeb4c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:09,372 DEBUG [RS:1;jenkins-hbase20:42249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@103720fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:09,372 DEBUG [RS:0;jenkins-hbase20:34333] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f91236a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-14 19:03:09,373 DEBUG [RS:1;jenkins-hbase20:42249] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1408b622, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-14 19:03:09,396 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase20:42249 2023-07-14 19:03:09,396 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:34333 2023-07-14 19:03:09,400 INFO [RS:0;jenkins-hbase20:34333] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 19:03:09,400 INFO [RS:1;jenkins-hbase20:42249] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 19:03:09,401 INFO [RS:0;jenkins-hbase20:34333] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 19:03:09,401 INFO [RS:1;jenkins-hbase20:42249] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 19:03:09,401 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 19:03:09,401 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 19:03:09,404 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38349,1689361387512 with isa=jenkins-hbase20.apache.org/148.251.75.209:42249, startcode=1689361388520 2023-07-14 19:03:09,404 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38349,1689361387512 with isa=jenkins-hbase20.apache.org/148.251.75.209:34333, startcode=1689361388479 2023-07-14 19:03:09,420 DEBUG [RS:0;jenkins-hbase20:34333] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 19:03:09,420 DEBUG [RS:1;jenkins-hbase20:42249] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 19:03:09,433 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-07-14 19:03:09,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-14 19:03:09,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-14 19:03:09,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-14 19:03:09,444 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-07-14 19:03:09,445 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-07-14 19:03:09,445 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,445 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-14 19:03:09,447 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,453 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1689361419453 2023-07-14 19:03:09,457 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-07-14 19:03:09,460 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 19:03:09,460 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-07-14 19:03:09,467 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 19:03:09,470 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-07-14 19:03:09,476 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59391, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 19:03:09,476 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47497, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 19:03:09,477 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-07-14 19:03:09,477 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-07-14 19:03:09,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-07-14 19:03:09,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-07-14 19:03:09,478 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,480 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-07-14 19:03:09,482 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-07-14 19:03:09,483 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-07-14 19:03:09,485 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-07-14 19:03:09,485 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-07-14 19:03:09,486 DEBUG [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38349] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2832) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:579) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:09,487 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689361389487,5,FailOnTimeoutGroup] 2023-07-14 19:03:09,487 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689361389487,5,FailOnTimeoutGroup] 2023-07-14 19:03:09,487 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,487 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-07-14 19:03:09,488 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,488 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,500 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38349] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,511 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 19:03:09,511 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b 2023-07-14 19:03:09,512 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:40151 2023-07-14 19:03:09,512 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 19:03:09,512 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-07-14 19:03:09,513 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b 2023-07-14 19:03:09,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:09,517 DEBUG [RS:1;jenkins-hbase20:42249] zookeeper.ZKUtil(162): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,518 WARN [RS:1;jenkins-hbase20:42249] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 19:03:09,523 INFO [RS:1;jenkins-hbase20:42249] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:09,524 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,524 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,42249,1689361388520] 2023-07-14 19:03:09,527 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(2830): Master is not running yet 2023-07-14 19:03:09,527 WARN [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1030): reportForDuty failed; sleeping 100 ms and then retrying. 2023-07-14 19:03:09,539 DEBUG [RS:1;jenkins-hbase20:42249] zookeeper.ZKUtil(162): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,549 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 19:03:09,555 INFO [RS:1;jenkins-hbase20:42249] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 19:03:09,572 INFO [RS:1;jenkins-hbase20:42249] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 19:03:09,575 INFO [RS:1;jenkins-hbase20:42249] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 19:03:09,576 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,577 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 19:03:09,583 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,583 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-14 19:03:09,584 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,584 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,584 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,584 DEBUG [RS:1;jenkins-hbase20:42249] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,585 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,585 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,585 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,597 INFO [RS:1;jenkins-hbase20:42249] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 19:03:09,599 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,42249,1689361388520-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,612 INFO [RS:1;jenkins-hbase20:42249] regionserver.Replication(203): jenkins-hbase20.apache.org,42249,1689361388520 started 2023-07-14 19:03:09,612 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,42249,1689361388520, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:42249, sessionid=0x1008cdd89820002 2023-07-14 19:03:09,612 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 19:03:09,612 DEBUG [RS:1;jenkins-hbase20:42249] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,612 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42249,1689361388520' 2023-07-14 19:03:09,613 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 19:03:09,613 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 19:03:09,614 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 19:03:09,614 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 19:03:09,614 DEBUG [RS:1;jenkins-hbase20:42249] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,614 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,42249,1689361388520' 2023-07-14 19:03:09,614 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 19:03:09,615 DEBUG [RS:1;jenkins-hbase20:42249] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 19:03:09,615 DEBUG [RS:1;jenkins-hbase20:42249] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 19:03:09,615 INFO [RS:1;jenkins-hbase20:42249] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 19:03:09,615 INFO [RS:1;jenkins-hbase20:42249] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 19:03:09,628 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38349,1689361387512 with isa=jenkins-hbase20.apache.org/148.251.75.209:34333, startcode=1689361388479 2023-07-14 19:03:09,629 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38349] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,630 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b 2023-07-14 19:03:09,630 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:40151 2023-07-14 19:03:09,630 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 19:03:09,633 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:09,633 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:09,634 DEBUG [RS:0;jenkins-hbase20:34333] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,634 WARN [RS:0;jenkins-hbase20:34333] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 19:03:09,634 INFO [RS:0;jenkins-hbase20:34333] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:09,634 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,34333,1689361388479] 2023-07-14 19:03:09,634 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,640 DEBUG [RS:0;jenkins-hbase20:34333] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,641 DEBUG [RS:0;jenkins-hbase20:34333] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:09,642 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 19:03:09,643 INFO [RS:0;jenkins-hbase20:34333] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 19:03:09,648 INFO [RS:0;jenkins-hbase20:34333] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 19:03:09,648 INFO [RS:0;jenkins-hbase20:34333] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 19:03:09,649 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,649 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 19:03:09,651 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,652 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,652 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,653 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,653 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,653 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,653 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-14 19:03:09,653 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,654 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,654 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,654 DEBUG [RS:0;jenkins-hbase20:34333] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:09,655 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,655 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,655 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,667 INFO [RS:0;jenkins-hbase20:34333] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 19:03:09,667 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,34333,1689361388479-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:09,678 INFO [RS:0;jenkins-hbase20:34333] regionserver.Replication(203): jenkins-hbase20.apache.org,34333,1689361388479 started 2023-07-14 19:03:09,678 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,34333,1689361388479, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:34333, sessionid=0x1008cdd89820001 2023-07-14 19:03:09,678 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 19:03:09,679 DEBUG [RS:0;jenkins-hbase20:34333] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,679 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34333,1689361388479' 2023-07-14 19:03:09,679 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 19:03:09,680 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 19:03:09,681 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 19:03:09,681 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 19:03:09,681 DEBUG [RS:0;jenkins-hbase20:34333] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:09,681 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,34333,1689361388479' 2023-07-14 19:03:09,681 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 19:03:09,682 DEBUG [RS:0;jenkins-hbase20:34333] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 19:03:09,682 DEBUG [RS:0;jenkins-hbase20:34333] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 19:03:09,682 INFO [RS:0;jenkins-hbase20:34333] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 19:03:09,683 INFO [RS:0;jenkins-hbase20:34333] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 19:03:09,724 INFO [RS:1;jenkins-hbase20:42249] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C42249%2C1689361388520, suffix=, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs, maxLogs=32 2023-07-14 19:03:09,743 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:09,743 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:09,747 INFO [RS:1;jenkins-hbase20:42249] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 2023-07-14 19:03:09,750 DEBUG [RS:1;jenkins-hbase20:42249] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK], DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK]] 2023-07-14 19:03:09,787 INFO [RS:0;jenkins-hbase20:34333] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34333%2C1689361388479, suffix=, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs, maxLogs=32 2023-07-14 19:03:09,808 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:09,808 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:09,813 INFO [RS:0;jenkins-hbase20:34333] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 2023-07-14 19:03:09,813 DEBUG [RS:0;jenkins-hbase20:34333] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK], DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK]] 2023-07-14 19:03:09,939 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:09,942 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 19:03:09,946 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info 2023-07-14 19:03:09,947 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 19:03:09,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:09,949 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 19:03:09,953 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/rep_barrier 2023-07-14 19:03:09,954 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 19:03:09,955 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:09,955 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 19:03:09,959 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table 2023-07-14 19:03:09,959 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 19:03:09,960 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:09,962 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:09,964 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:09,968 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 19:03:09,970 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 19:03:09,974 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 19:03:09,974 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60218807, jitterRate=-0.1026698499917984}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 19:03:09,975 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 19:03:09,975 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 19:03:09,975 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 19:03:09,975 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 19:03:09,975 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 19:03:09,975 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 19:03:09,976 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 19:03:09,976 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 19:03:09,982 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-07-14 19:03:09,982 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-07-14 19:03:09,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 19:03:10,000 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 19:03:10,003 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-07-14 19:03:10,155 DEBUG [jenkins-hbase20:38349] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=2, allServersCount=2 2023-07-14 19:03:10,161 DEBUG [jenkins-hbase20:38349] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-14 19:03:10,167 DEBUG [jenkins-hbase20:38349] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 19:03:10,167 DEBUG [jenkins-hbase20:38349] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 19:03:10,167 DEBUG [jenkins-hbase20:38349] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 19:03:10,169 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34333,1689361388479, state=OPENING 2023-07-14 19:03:10,175 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-07-14 19:03:10,176 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:10,176 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 19:03:10,179 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34333,1689361388479}] 2023-07-14 19:03:10,364 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:10,366 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 19:03:10,370 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44354, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 19:03:10,382 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 19:03:10,382 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:10,386 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs, maxLogs=32 2023-07-14 19:03:10,402 DEBUG [RS-EventLoopGroup-4-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:10,402 DEBUG [RS-EventLoopGroup-4-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:10,408 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta 2023-07-14 19:03:10,409 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK], DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK]] 2023-07-14 19:03:10,409 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:10,411 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 19:03:10,427 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 19:03:10,432 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 19:03:10,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 19:03:10,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:10,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 19:03:10,437 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 19:03:10,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 19:03:10,443 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info 2023-07-14 19:03:10,443 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info 2023-07-14 19:03:10,444 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 19:03:10,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:10,445 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 19:03:10,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/rep_barrier 2023-07-14 19:03:10,447 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/rep_barrier 2023-07-14 19:03:10,448 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 19:03:10,449 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:10,449 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 19:03:10,450 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table 2023-07-14 19:03:10,450 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table 2023-07-14 19:03:10,451 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 19:03:10,452 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:10,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:10,456 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:10,459 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 19:03:10,462 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 19:03:10,463 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64432702, jitterRate=-0.03987792134284973}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 19:03:10,463 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 19:03:10,474 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1689361390358 2023-07-14 19:03:10,492 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 19:03:10,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 19:03:10,497 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,34333,1689361388479, state=OPEN 2023-07-14 19:03:10,500 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 19:03:10,500 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 19:03:10,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-07-14 19:03:10,507 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,34333,1689361388479 in 321 msec 2023-07-14 19:03:10,513 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-07-14 19:03:10,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 517 msec 2023-07-14 19:03:10,519 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1460 sec 2023-07-14 19:03:10,520 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1689361390520, completionTime=-1 2023-07-14 19:03:10,520 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2023-07-14 19:03:10,520 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-07-14 19:03:10,578 DEBUG [hconnection-0x5ab972ed-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:10,581 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44366, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:10,615 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=2 2023-07-14 19:03:10,615 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1689361450615 2023-07-14 19:03:10,615 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1689361510615 2023-07-14 19:03:10,615 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 95 msec 2023-07-14 19:03:10,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38349,1689361387512-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:10,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38349,1689361387512-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:10,640 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38349,1689361387512-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:10,641 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:38349, period=300000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:10,642 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:10,648 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-07-14 19:03:10,655 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-07-14 19:03:10,656 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-07-14 19:03:10,665 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-07-14 19:03:10,668 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 19:03:10,670 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 19:03:10,690 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:10,692 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 empty. 2023-07-14 19:03:10,693 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:10,694 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-07-14 19:03:10,727 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-07-14 19:03:10,730 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => c5039524108f6f5f0d888c4d54df6bc4, NAME => 'hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp 2023-07-14 19:03:10,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:10,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing c5039524108f6f5f0d888c4d54df6bc4, disabling compactions & flushes 2023-07-14 19:03:10,746 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:10,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:10,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. after waiting 0 ms 2023-07-14 19:03:10,746 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:10,746 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:10,747 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for c5039524108f6f5f0d888c4d54df6bc4: 2023-07-14 19:03:10,752 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 19:03:10,767 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689361390755"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689361390755"}]},"ts":"1689361390755"} 2023-07-14 19:03:10,791 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 19:03:10,793 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 19:03:10,797 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689361390793"}]},"ts":"1689361390793"} 2023-07-14 19:03:10,801 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-07-14 19:03:10,805 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-14 19:03:10,807 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 19:03:10,807 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 19:03:10,807 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 19:03:10,810 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN}] 2023-07-14 19:03:10,814 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN 2023-07-14 19:03:10,816 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,34333,1689361388479; forceNewPlan=false, retain=false 2023-07-14 19:03:10,970 INFO [jenkins-hbase20:38349] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 19:03:10,971 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c5039524108f6f5f0d888c4d54df6bc4, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:10,971 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689361390971"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689361390971"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689361390971"}]},"ts":"1689361390971"} 2023-07-14 19:03:10,975 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure c5039524108f6f5f0d888c4d54df6bc4, server=jenkins-hbase20.apache.org,34333,1689361388479}] 2023-07-14 19:03:11,137 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:11,141 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c5039524108f6f5f0d888c4d54df6bc4, NAME => 'hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:11,142 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,142 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:11,142 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,142 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,146 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,149 DEBUG [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info 2023-07-14 19:03:11,149 DEBUG [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info 2023-07-14 19:03:11,149 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c5039524108f6f5f0d888c4d54df6bc4 columnFamilyName info 2023-07-14 19:03:11,150 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] regionserver.HStore(310): Store=c5039524108f6f5f0d888c4d54df6bc4/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:11,152 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,153 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,158 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:11,161 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 19:03:11,162 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c5039524108f6f5f0d888c4d54df6bc4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59983955, jitterRate=-0.10616941750049591}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:11,162 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c5039524108f6f5f0d888c4d54df6bc4: 2023-07-14 19:03:11,164 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4., pid=6, masterSystemTime=1689361391129 2023-07-14 19:03:11,167 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:11,167 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:11,168 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=c5039524108f6f5f0d888c4d54df6bc4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:11,168 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689361391167"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689361391167"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689361391167"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689361391167"}]},"ts":"1689361391167"} 2023-07-14 19:03:11,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-07-14 19:03:11,174 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure c5039524108f6f5f0d888c4d54df6bc4, server=jenkins-hbase20.apache.org,34333,1689361388479 in 196 msec 2023-07-14 19:03:11,178 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-07-14 19:03:11,179 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN in 364 msec 2023-07-14 19:03:11,180 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 19:03:11,180 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689361391180"}]},"ts":"1689361391180"} 2023-07-14 19:03:11,183 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-07-14 19:03:11,186 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 19:03:11,189 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 528 msec 2023-07-14 19:03:11,270 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-07-14 19:03:11,273 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-07-14 19:03:11,273 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:11,310 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-07-14 19:03:11,326 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 19:03:11,332 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-07-14 19:03:11,344 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-07-14 19:03:11,356 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-07-14 19:03:11,361 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-07-14 19:03:11,374 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-07-14 19:03:11,376 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-07-14 19:03:11,377 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.799sec 2023-07-14 19:03:11,380 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-07-14 19:03:11,382 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-07-14 19:03:11,383 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-07-14 19:03:11,384 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38349,1689361387512-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-07-14 19:03:11,385 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38349,1689361387512-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-07-14 19:03:11,394 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-07-14 19:03:11,450 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(139): Connect 0x081026c2 to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:11,455 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5057acd6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:11,468 DEBUG [hconnection-0x9dd1498-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:11,479 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44382, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:11,488 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:11,497 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-07-14 19:03:11,499 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-07-14 19:03:11,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 't', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-07-14 19:03:11,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=t 2023-07-14 19:03:11,515 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_PRE_OPERATION 2023-07-14 19:03:11,517 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-07-14 19:03:11,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.MasterRpcServices(700): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "t" procId is: 9 2023-07-14 19:03:11,520 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,521 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 empty. 2023-07-14 19:03:11,522 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,522 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived t regions 2023-07-14 19:03:11,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 19:03:11,545 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp/data/default/t/.tabledesc/.tableinfo.0000000001 2023-07-14 19:03:11,547 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(7675): creating {ENCODED => f8ff06c91c03cb8152ab689acbbf13c3, NAME => 't,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='t', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'f', BLOOMFILTER => 'NONE', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/.tmp 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(866): Instantiated t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1604): Closing f8ff06c91c03cb8152ab689acbbf13c3, disabling compactions & flushes 2023-07-14 19:03:11,563 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1626): Closing region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1714): Acquired close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. after waiting 0 ms 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1724): Updates disabled for region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,563 INFO [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1838): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,563 DEBUG [RegionOpenAndInit-t-pool-0] regionserver.HRegion(1558): Region close journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:11,568 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ADD_TO_META 2023-07-14 19:03:11,569 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361391569"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1689361391569"}]},"ts":"1689361391569"} 2023-07-14 19:03:11,572 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-07-14 19:03:11,574 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-07-14 19:03:11,575 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689361391574"}]},"ts":"1689361391574"} 2023-07-14 19:03:11,577 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLING in hbase:meta 2023-07-14 19:03:11,581 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase20.apache.org=0} racks are {/default-rack=0} 2023-07-14 19:03:11,582 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-07-14 19:03:11,582 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-07-14 19:03:11,582 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-07-14 19:03:11,582 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN}] 2023-07-14 19:03:11,584 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN 2023-07-14 19:03:11,586 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,42249,1689361388520; forceNewPlan=false, retain=false 2023-07-14 19:03:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 19:03:11,736 INFO [jenkins-hbase20:38349] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-07-14 19:03:11,738 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:11,739 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361391738"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689361391738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689361391738"}]},"ts":"1689361391738"} 2023-07-14 19:03:11,745 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,42249,1689361388520}] 2023-07-14 19:03:11,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 19:03:11,902 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:11,903 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 19:03:11,909 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54064, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 19:03:11,921 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8ff06c91c03cb8152ab689acbbf13c3, NAME => 't,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:11,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:11,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,925 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,927 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:11,927 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:11,928 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8ff06c91c03cb8152ab689acbbf13c3 columnFamilyName f 2023-07-14 19:03:11,929 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(310): Store=f8ff06c91c03cb8152ab689acbbf13c3/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:11,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:11,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-07-14 19:03:11,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f8ff06c91c03cb8152ab689acbbf13c3; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58914579, jitterRate=-0.12210436165332794}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:11,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:11,943 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3., pid=11, masterSystemTime=1689361391901 2023-07-14 19:03:11,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:11,950 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:11,950 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361391950"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689361391950"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689361391950"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689361391950"}]},"ts":"1689361391950"} 2023-07-14 19:03:11,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-07-14 19:03:11,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,42249,1689361388520 in 209 msec 2023-07-14 19:03:11,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-07-14 19:03:11,961 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN in 376 msec 2023-07-14 19:03:11,962 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-07-14 19:03:11,963 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"t","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1689361391962"}]},"ts":"1689361391962"} 2023-07-14 19:03:11,965 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=t, state=ENABLED in hbase:meta 2023-07-14 19:03:11,967 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=t execute state=CREATE_TABLE_POST_OPERATION 2023-07-14 19:03:11,969 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=t in 456 msec 2023-07-14 19:03:12,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.MasterRpcServices(1230): Checking to see if procedure is done pid=9 2023-07-14 19:03:12,146 INFO [Listener at localhost.localdomain/40851] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:t, procId: 9 completed 2023-07-14 19:03:12,147 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3430): Waiting until all regions of table t get assigned. Timeout = 60000ms 2023-07-14 19:03:12,150 INFO [Listener at localhost.localdomain/40851] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 19:03:12,155 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3484): All regions for table t assigned to meta. Checking AM states. 2023-07-14 19:03:12,155 INFO [Listener at localhost.localdomain/40851] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-07-14 19:03:12,155 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3504): All regions for table t assigned. 2023-07-14 19:03:12,159 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:12,162 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54078, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:12,211 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686a Thread=293, OpenFileDescriptor=585, MaxFileDescriptor=60000, SystemLoadAverage=143, ProcessCount=165, AvailableMemoryMB=5289 2023-07-14 19:03:12,213 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(134): START ************ TEST3686A---1 2023-07-14 19:03:12,216 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 19:03:12,218 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,218 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(136): START ************ TEST3686A---1111 2023-07-14 19:03:12,218 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(140): ************ TEST3686A 2023-07-14 19:03:12,222 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace/table:state/1689361391180/Put/vlen=2/seqid=0} 2023-07-14 19:03:12,222 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=hbase:namespace, state=ENABLED 2023-07-14 19:03:12,222 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:regioninfo/1689361391167/Put/vlen=41/seqid=0, hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:seqnumDuringOpen/1689361391167/Put/vlen=8/seqid=0, hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:server/1689361391167/Put/vlen=32/seqid=0, hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:serverstartcode/1689361391167/Put/vlen=8/seqid=0, hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:sn/1689361390971/Put/vlen=46/seqid=0, hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4./info:state/1689361391167/Put/vlen=4/seqid=0} 2023-07-14 19:03:12,223 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => c5039524108f6f5f0d888c4d54df6bc4, NAME => 'hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:12,223 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t/table:state/1689361391962/Put/vlen=2/seqid=0} 2023-07-14 19:03:12,223 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(680): fullScanMetaAndPrint.Table State={}tableName=t, state=ENABLED 2023-07-14 19:03:12,223 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(677): fullScanMetaAndPrint.Current Meta Row: keyvalues={t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:regioninfo/1689361391950/Put/vlen=35/seqid=0, t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:seqnumDuringOpen/1689361391950/Put/vlen=8/seqid=0, t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:server/1689361391950/Put/vlen=32/seqid=0, t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:serverstartcode/1689361391950/Put/vlen=8/seqid=0, t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:sn/1689361391738/Put/vlen=46/seqid=0, t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3./info:state/1689361391950/Put/vlen=4/seqid=0} 2023-07-14 19:03:12,223 INFO [Listener at localhost.localdomain/40851] hbase.MetaTableAccessor(688): fullScanMetaAndPrint.HRI Print={ENCODED => f8ff06c91c03cb8152ab689acbbf13c3, NAME => 't,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:12,224 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(139): Connect 0x7c979b9c to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:12,229 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46db8ca, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:12,229 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(150): START ************ TEST3686A---22 2023-07-14 19:03:12,230 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(153): START ************ TEST3686A---33 2023-07-14 19:03:12,234 DEBUG [hconnection-0x61794e34-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:12,239 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44392, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:12,250 DEBUG [hconnection-0x61794e34-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:12,253 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:54090, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:12,255 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(157): START ************ TEST3686A---44 2023-07-14 19:03:12,259 ERROR [Listener at localhost.localdomain/40851] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,42249,1689361388520: die! ***** 2023-07-14 19:03:12,259 ERROR [Listener at localhost.localdomain/40851] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 19:03:12,262 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 19:03:12,267 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 19:03:12,267 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 19:03:12,268 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 19:03:12,269 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 904921088, "init": 524288000, "max": 2051014656, "used": 411406544 }, "NonHeapMemoryUsage": { "committed": 113205248, "init": 2555904, "max": -1, "used": 111280912 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org" } ] } 2023-07-14 19:03:12,274 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38349] master.MasterRpcServices(612): jenkins-hbase20.apache.org,42249,1689361388520 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,42249,1689361388520: die! ***** 2023-07-14 19:03:12,278 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,42249,1689361388520' ***** 2023-07-14 19:03:12,278 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 19:03:12,279 INFO [RS:1;jenkins-hbase20:42249] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 19:03:12,279 INFO [RS:1;jenkins-hbase20:42249] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 19:03:12,279 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 19:03:12,279 INFO [RS:1;jenkins-hbase20:42249] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 19:03:12,279 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(3305): Received CLOSE for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] regionserver.RSRpcServices(3499): Server shutting down and client tried to access missing scanner -5798845735302594559 2023-07-14 19:03:12,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:12,281 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:12,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f8ff06c91c03cb8152ab689acbbf13c3, disabling compactions & flushes 2023-07-14 19:03:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 43 connection: 148.251.75.209:54090 deadline: 1689361452281, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting 2023-07-14 19:03:12,282 DEBUG [RS:1;jenkins-hbase20:42249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ef0dd52 to 127.0.0.1:51669 2023-07-14 19:03:12,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,282 DEBUG [RS:1;jenkins-hbase20:42249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:12,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. after waiting 0 ms 2023-07-14 19:03:12,283 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,283 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1474): Waiting on 1 regions to close 2023-07-14 19:03:12,283 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1478): Online Regions={f8ff06c91c03cb8152ab689acbbf13c3=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.} 2023-07-14 19:03:12,283 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 270 in region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,284 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:12,284 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:12,285 DEBUG [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1504): Waiting on 2023-07-14 19:03:12,286 DEBUG [Listener at localhost.localdomain/40851] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause org.apache.hadoop.hbase.DoNotRetryIOException: Resetting the scanner -- see exception cause at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:232) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) ... 11 more Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException): org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 19:03:12,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:12,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:54090 deadline: 1689361452290, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting 2023-07-14 19:03:12,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:12,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42249] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Scan size: 125 connection: 148.251.75.209:54090 deadline: 1689361452404, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,42249,1689361388520 aborting 2023-07-14 19:03:12,485 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,42249,1689361388520; all regions closed. 2023-07-14 19:03:12,504 DEBUG [RS:1;jenkins-hbase20:42249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:12,505 INFO [RS:1;jenkins-hbase20:42249] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:12,505 INFO [RS:1;jenkins-hbase20:42249] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-07-14 19:03:12,506 INFO [RS:1;jenkins-hbase20:42249] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 19:03:12,506 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 19:03:12,506 INFO [RS:1;jenkins-hbase20:42249] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 19:03:12,506 INFO [RS:1;jenkins-hbase20:42249] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 19:03:12,507 INFO [RS:1;jenkins-hbase20:42249] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:42249 2023-07-14 19:03:12,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:12,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:12,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:12,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:12,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:12,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:12,519 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,42249,1689361388520] 2023-07-14 19:03:12,519 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,42249,1689361388520 znode expired, triggering replicatorRemoved event 2023-07-14 19:03:12,519 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,42249,1689361388520; numProcessing=1 2023-07-14 19:03:12,521 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,42249,1689361388520 already deleted, retry=false 2023-07-14 19:03:12,521 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,42249,1689361388520 on jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:12,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:12,528 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,42249,1689361388520, splitWal=true, meta=false 2023-07-14 19:03:12,528 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=12 for jenkins-hbase20.apache.org,42249,1689361388520 (carryingMeta=false) jenkins-hbase20.apache.org,42249,1689361388520/CRASHED/regionCount=1/lock=java.util.concurrent.locks.ReentrantReadWriteLock@13ab9b8c[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 19:03:12,531 INFO [PEWorker-5] procedure.ServerCrashProcedure(161): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,42249,1689361388520, splitWal=true, meta=false 2023-07-14 19:03:12,533 INFO [PEWorker-5] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,42249,1689361388520 had 1 regions 2023-07-14 19:03:12,534 INFO [PEWorker-5] procedure.ServerCrashProcedure(300): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,42249,1689361388520, splitWal=true, meta=false, isMeta: false 2023-07-14 19:03:12,536 DEBUG [PEWorker-5] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting 2023-07-14 19:03:12,538 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,42249,1689361388520 WAL count=1, meta=false 2023-07-14 19:03:12,541 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726}] 2023-07-14 19:03:12,550 DEBUG [PEWorker-4] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:12,552 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, worker=jenkins-hbase20.apache.org,34333,1689361388479}] 2023-07-14 19:03:12,586 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:12,618 WARN [RPCClient-NioEventLoopGroup-5-5] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:42249 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42249 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:12,618 DEBUG [RPCClient-NioEventLoopGroup-5-5] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:42249 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:42249 2023-07-14 19:03:12,619 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:12,620 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:42249-0x1008cdd89820002, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:12,620 INFO [RS:1;jenkins-hbase20:42249] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,42249,1689361388520; zookeeper connection closed. 2023-07-14 19:03:12,620 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@366198c8] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@366198c8 2023-07-14 19:03:12,720 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34333] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=14 2023-07-14 19:03:12,742 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, size=1.1 K (1149bytes) 2023-07-14 19:03:12,743 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 2023-07-14 19:03:12,747 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 after 4ms 2023-07-14 19:03:12,752 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 took 11ms 2023-07-14 19:03:12,766 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for f8ff06c91c03cb8152ab689acbbf13c3: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "f" sequence_id: 3 } 2023-07-14 19:03:12,768 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 so closing down 2023-07-14 19:03:12,768 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 19:03:12,773 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] util.CommonFSUtils$DfsBuilderUtility(753): Could not find replicate method on builder; will not set replicate when creating output stream java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.replicate() at java.lang.Class.getMethod(Class.java:1786) at org.apache.hadoop.hbase.util.CommonFSUtils$DfsBuilderUtility.(CommonFSUtils.java:750) at org.apache.hadoop.hbase.util.CommonFSUtils.createForWal(CommonFSUtils.java:802) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.initOutput(ProtobufLogWriter.java:102) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:160) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:78) at org.apache.hadoop.hbase.wal.FSHLogProvider.createWriter(FSHLogProvider.java:62) at org.apache.hadoop.hbase.wal.WALFactory.createRecoveredEditsWriter(WALFactory.java:402) at org.apache.hadoop.hbase.wal.WALSplitter.createWriter(WALSplitter.java:516) at org.apache.hadoop.hbase.wal.AbstractRecoveredEditsOutputSink.createRecoveredEditsWriter(AbstractRecoveredEditsOutputSink.java:69) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.getRecoveredEditsWriter(RecoveredEditsOutputSink.java:81) at org.apache.hadoop.hbase.wal.RecoveredEditsOutputSink.append(RecoveredEditsOutputSink.java:63) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.writeBuffer(OutputSink.java:249) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.doRun(OutputSink.java:241) at org.apache.hadoop.hbase.wal.OutputSink$WriterThread.run(OutputSink.java:211) 2023-07-14 19:03:12,776 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp 2023-07-14 19:03:12,777 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 19:03:12,814 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp (wrote 10 edits, skipped 0 edits in 0 ms) 2023-07-14 19:03:12,817 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000013 2023-07-14 19:03:12,817 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 11 edits across 1 Regions in 64 ms; skipped=1; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, size=1.1 K, length=1149, corrupted=false, cancelled=false 2023-07-14 19:03:12,817 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, journal: Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, size=1.1 K (1149bytes) at 1689361392742Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 so closing down at 1689361392768 (+26 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp at 1689361392776 (+8 ms)3 split writer threads finished at 1689361392777 (+1 ms)Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp (wrote 10 edits, skipped 0 edits in 0 ms) at 1689361392814 (+37 ms)Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000013 at 1689361392817 (+3 ms)Processed 11 edits across 1 Regions in 64 ms; skipped=1; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, size=1.1 K, length=1149, corrupted=false, cancelled=false at 1689361392817 2023-07-14 19:03:12,818 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 2023-07-14 19:03:12,821 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=14 2023-07-14 19:03:12,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.HMaster(3641): Remote procedure done, pid=14 2023-07-14 19:03:12,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting/jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726 to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs 2023-07-14 19:03:12,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-07-14 19:03:12,835 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, worker=jenkins-hbase20.apache.org,34333,1689361388479 in 280 msec 2023-07-14 19:03:12,836 DEBUG [PEWorker-3] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:12,841 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-07-14 19:03:12,841 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C42249%2C1689361388520.1689361389726, worker=jenkins-hbase20.apache.org,34333,1689361388479 in 298 msec 2023-07-14 19:03:12,844 INFO [PEWorker-5] master.SplitLogManager(171): hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting dir is empty, no logs to split. 2023-07-14 19:03:12,844 INFO [PEWorker-5] master.SplitWALManager(106): jenkins-hbase20.apache.org,42249,1689361388520 WAL count=0, meta=false 2023-07-14 19:03:12,844 DEBUG [PEWorker-5] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,42249,1689361388520 WAL splitting is done? wals=0, meta=false 2023-07-14 19:03:12,847 INFO [PEWorker-5] procedure.ServerCrashProcedure(282): Remove WAL directory for jenkins-hbase20.apache.org,42249,1689361388520 failed, ignore...File hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,42249,1689361388520-splitting does not exist. 2023-07-14 19:03:12,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN}] 2023-07-14 19:03:12,850 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN 2023-07-14 19:03:12,852 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 19:03:12,925 DEBUG [hconnection-0x61794e34-shared-pool-2] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:42249 this server is in the failed servers list 2023-07-14 19:03:13,003 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:13,003 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361393002"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689361393002"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689361393002"}]},"ts":"1689361393002"} 2023-07-14 19:03:13,010 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=15, state=RUNNABLE; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,34333,1689361388479}] 2023-07-14 19:03:13,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:13,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8ff06c91c03cb8152ab689acbbf13c3, NAME => 't,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:13,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:13,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,172 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,174 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:13,174 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:13,175 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8ff06c91c03cb8152ab689acbbf13c3 columnFamilyName f 2023-07-14 19:03:13,175 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(310): Store=f8ff06c91c03cb8152ab689acbbf13c3/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:13,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000013 2023-07-14 19:03:13,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 10, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=13, path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000013 2023-07-14 19:03:13,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing f8ff06c91c03cb8152ab689acbbf13c3 1/1 column families, dataSize=270 B heapSize=1.27 KB 2023-07-14 19:03:13,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=270 B at sequenceid=13 (bloomFilter=false), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/.tmp/f/54c410394dc8478abfdf0a46bcfa0679 2023-07-14 19:03:13,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/.tmp/f/54c410394dc8478abfdf0a46bcfa0679 as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f/54c410394dc8478abfdf0a46bcfa0679 2023-07-14 19:03:13,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f/54c410394dc8478abfdf0a46bcfa0679, entries=10, sequenceid=13, filesize=4.8 K 2023-07-14 19:03:13,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~270 B/270, heapSize ~1.25 KB/1280, currentSize=0 B/0 for f8ff06c91c03cb8152ab689acbbf13c3 in 93ms, sequenceid=13, compaction requested=false; wal=null 2023-07-14 19:03:13,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 't' 2023-07-14 19:03:13,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/0000000000000000013 2023-07-14 19:03:13,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:13,296 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=1 2023-07-14 19:03:13,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f8ff06c91c03cb8152ab689acbbf13c3; next sequenceid=14; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67409388, jitterRate=0.004478156566619873}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:13,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:13,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3., pid=16, masterSystemTime=1689361393164 2023-07-14 19:03:13,301 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:13,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:13,302 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPEN, openSeqNum=14, regionLocation=jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:13,303 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361393302"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689361393302"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689361393302"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689361393302"}]},"ts":"1689361393302"} 2023-07-14 19:03:13,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=15 2023-07-14 19:03:13,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=15, state=SUCCESS; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,34333,1689361388479 in 296 msec 2023-07-14 19:03:13,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-07-14 19:03:13,312 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN in 460 msec 2023-07-14 19:03:13,312 INFO [PEWorker-5] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,42249,1689361388520 after splitting done 2023-07-14 19:03:13,312 DEBUG [PEWorker-5] master.DeadServer(114): Removed jenkins-hbase20.apache.org,42249,1689361388520 from processing; numProcessing=0 2023-07-14 19:03:13,314 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,42249,1689361388520, splitWal=true, meta=false in 790 msec 2023-07-14 19:03:13,456 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7c979b9c to 127.0.0.1:51669 2023-07-14 19:03:13,457 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:13,457 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(168): ************ END TEST3686A 2023-07-14 19:03:13,473 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686a Thread=270 (was 293), OpenFileDescriptor=567 (was 585), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=139 (was 143), ProcessCount=165 (was 165), AvailableMemoryMB=5215 (was 5289) 2023-07-14 19:03:13,483 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test3686b Thread=270, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=139, ProcessCount=165, AvailableMemoryMB=5215 2023-07-14 19:03:13,484 INFO [Listener at localhost.localdomain/40851] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:42249 2023-07-14 19:03:13,499 INFO [Listener at localhost.localdomain/40851] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-07-14 19:03:13,499 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:13,500 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:13,500 INFO [Listener at localhost.localdomain/40851] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-07-14 19:03:13,500 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-07-14 19:03:13,500 INFO [Listener at localhost.localdomain/40851] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-07-14 19:03:13,500 INFO [Listener at localhost.localdomain/40851] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-07-14 19:03:13,502 INFO [Listener at localhost.localdomain/40851] ipc.NettyRpcServer(120): Bind to /148.251.75.209:38019 2023-07-14 19:03:13,502 INFO [Listener at localhost.localdomain/40851] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-07-14 19:03:13,505 DEBUG [Listener at localhost.localdomain/40851] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-07-14 19:03:13,505 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:13,506 INFO [Listener at localhost.localdomain/40851] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-07-14 19:03:13,507 INFO [Listener at localhost.localdomain/40851] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38019 connecting to ZooKeeper ensemble=127.0.0.1:51669 2023-07-14 19:03:13,510 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:380190x0, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-07-14 19:03:13,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(625): regionserver:38019-0x1008cdd89820008 connected 2023-07-14 19:03:13,512 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-07-14 19:03:13,513 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2023-07-14 19:03:13,514 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ZKUtil(164): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-07-14 19:03:13,515 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38019 2023-07-14 19:03:13,515 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38019 2023-07-14 19:03:13,515 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38019 2023-07-14 19:03:13,516 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38019 2023-07-14 19:03:13,516 DEBUG [Listener at localhost.localdomain/40851] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38019 2023-07-14 19:03:13,518 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(951): ClusterId : cfbfac95-a50f-45d8-a4bb-94c79ffe7eeb 2023-07-14 19:03:13,518 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-07-14 19:03:13,525 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-07-14 19:03:13,525 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-07-14 19:03:13,530 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-07-14 19:03:13,532 DEBUG [RS:2;jenkins-hbase20:38019] zookeeper.ReadOnlyZKClient(139): Connect 0x4fa40606 to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:13,540 DEBUG [RS:2;jenkins-hbase20:38019] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4163e182, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:13,541 DEBUG [RS:2;jenkins-hbase20:38019] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@455c6d76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-14 19:03:13,548 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase20:38019 2023-07-14 19:03:13,548 INFO [RS:2;jenkins-hbase20:38019] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-07-14 19:03:13,548 INFO [RS:2;jenkins-hbase20:38019] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-07-14 19:03:13,548 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1022): About to register with Master. 2023-07-14 19:03:13,549 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(2811): reportForDuty to master=jenkins-hbase20.apache.org,38349,1689361387512 with isa=jenkins-hbase20.apache.org/148.251.75.209:38019, startcode=1689361393499 2023-07-14 19:03:13,549 DEBUG [RS:2;jenkins-hbase20:38019] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-07-14 19:03:13,554 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42593, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-07-14 19:03:13,555 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38349] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,555 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1595): Config from master: hbase.rootdir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b 2023-07-14 19:03:13,556 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1595): Config from master: fs.defaultFS=hdfs://localhost.localdomain:40151 2023-07-14 19:03:13,556 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1595): Config from master: hbase.master.info.port=-1 2023-07-14 19:03:13,557 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:13,557 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:13,558 DEBUG [RS:2;jenkins-hbase20:38019] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,558 WARN [RS:2;jenkins-hbase20:38019] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-07-14 19:03:13,558 INFO [RS:2;jenkins-hbase20:38019] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:13,558 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,38019,1689361393499] 2023-07-14 19:03:13,558 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1948): logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:13,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,565 DEBUG [RS:2;jenkins-hbase20:38019] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:13,565 DEBUG [RS:2;jenkins-hbase20:38019] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,566 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-07-14 19:03:13,566 INFO [RS:2;jenkins-hbase20:38019] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-07-14 19:03:13,568 INFO [RS:2;jenkins-hbase20:38019] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-07-14 19:03:13,568 INFO [RS:2;jenkins-hbase20:38019] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-07-14 19:03:13,568 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,569 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer$CompactionChecker(1837): CompactionChecker runs every PT1S 2023-07-14 19:03:13,570 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,570 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,570 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,570 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,571 DEBUG [RS:2;jenkins-hbase20:38019] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-07-14 19:03:13,573 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,573 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,573 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,584 INFO [RS:2;jenkins-hbase20:38019] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-07-14 19:03:13,584 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,38019,1689361393499-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-07-14 19:03:13,598 INFO [RS:2;jenkins-hbase20:38019] regionserver.Replication(203): jenkins-hbase20.apache.org,38019,1689361393499 started 2023-07-14 19:03:13,598 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1637): Serving as jenkins-hbase20.apache.org,38019,1689361393499, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:38019, sessionid=0x1008cdd89820008 2023-07-14 19:03:13,598 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3254): Started new server=Thread[RS:2;jenkins-hbase20:38019,5,FailOnTimeoutGroup] 2023-07-14 19:03:13,598 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-07-14 19:03:13,598 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:13,598 DEBUG [RS:2;jenkins-hbase20:38019] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,598 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(177): START ************ test3686b 2023-07-14 19:03:13,598 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38019,1689361393499' 2023-07-14 19:03:13,598 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-07-14 19:03:13,599 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-07-14 19:03:13,599 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,38019,1689361393499' 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-07-14 19:03:13,600 DEBUG [RS:2;jenkins-hbase20:38019] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-07-14 19:03:13,600 INFO [RS:2;jenkins-hbase20:38019] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-07-14 19:03:13,601 INFO [RS:2;jenkins-hbase20:38019] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-07-14 19:03:13,602 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 19:03:13,602 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:13,603 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(139): Connect 0x7d468e67 to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:13,611 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@354a9fd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:13,614 DEBUG [hconnection-0x6a678530-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:13,618 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:44404, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:13,707 INFO [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38019%2C1689361393499, suffix=, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,38019,1689361393499, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs, maxLogs=32 2023-07-14 19:03:13,728 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:13,728 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:13,731 INFO [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,38019,1689361393499/jenkins-hbase20.apache.org%2C38019%2C1689361393499.1689361393711 2023-07-14 19:03:13,732 DEBUG [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK], DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK]] 2023-07-14 19:03:15,958 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-07-14 19:03:16,036 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-07-14 19:03:16,037 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-07-14 19:03:17,647 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 19:03:17,647 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-07-14 19:03:28,624 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.RSRpcServices$ScannerListener(511): Scanner lease 7142323604414267411 expired clientIPAndPort=148.251.75.209:44404, userName=jenkins, regionInfo=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34333] regionserver.RSRpcServices(3094): Client tried to access missing scanner 7142323604414267411 2023-07-14 19:03:30,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34333] ipc.CallRunner(144): callId: 2 service: ClientService methodName: Scan size: 42 connection: 148.251.75.209:44404 deadline: 1689361470627, exception=org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7142323604414267411'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. 2023-07-14 19:03:30,632 DEBUG [Listener at localhost.localdomain/40851] client.ScannerCallableWithReplicas(204): Scan with primary region returns org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7142323604414267411'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7142323604414267411'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:376) at org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:364) at org.apache.hadoop.hbase.client.ScannerCallable.next(ScannerCallable.java:194) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:258) at org.apache.hadoop.hbase.client.ScannerCallable.rpcCall(ScannerCallable.java:58) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:124) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:187) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:393) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:74) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException: Unknown scanner '7142323604414267411'. This can happen due to any of the following reasons: a) Scanner id given is wrong, b) Scanner lease expired because of long wait between consecutive client checkins, c) Server may be closing down, d) RegionServer restart during upgrade. If the issue is due to reason (b), a possible fix would be increasing the value of'hbase.client.scanner.timeout.period' configuration. at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionScanner(RSRpcServices.java:3095) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3528) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ... 1 more 2023-07-14 19:03:30,641 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d468e67 to 127.0.0.1:51669 2023-07-14 19:03:30,641 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:30,641 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(201): END ************ END test3686b 2023-07-14 19:03:30,654 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test3686b Thread=293 (was 270) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x4fa40606 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/719575305.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-74110187-148.251.75.209-1689361384905:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-794209219_17 at /127.0.0.1:49172 [Receiving block BP-74110187-148.251.75.209-1689361384905:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-794209219_17 at /127.0.0.1:34386 [Receiving block BP-74110187-148.251.75.209-1689361384905:blk_1073741841_1017] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:209) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:211) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:528) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:968) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:877) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase20:38019-longCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: jenkins-hbase20:38019Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.getCallRunner(RpcExecutor.java:340) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x7d468e67-SendThread(127.0.0.1:51669) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x7d468e67 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529) org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512) org.apache.zookeeper.ClientCnxn.close(ClientCnxn.java:1481) org.apache.zookeeper.ZooKeeper.close(ZooKeeper.java:1415) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.closeZk(ReadOnlyZKClient.java:304) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:352) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$46/719575305.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x4fa40606-SendThread(127.0.0.1:51669) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x4fa40606-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RS:2;jenkins-hbase20:38019 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1092) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:175) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:123) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:159) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1873) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:319) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:156) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: AsyncFSWAL-0-hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b-prefix:jenkins-hbase20.apache.org,38019,1689361393499 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-6-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:302) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:366) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: PacketResponder: BP-74110187-148.251.75.209-1689361384905:blk_1073741841_1017, type=LAST_IN_PIPELINE java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1327) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1399) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-5-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:51669@0x7d468e67-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost.localdomain:40151 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=38019 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:105) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) - Thread LEAK? -, OpenFileDescriptor=571 (was 567) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=115 (was 139), ProcessCount=165 (was 165), AvailableMemoryMB=5200 (was 5215) 2023-07-14 19:03:30,667 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(147): before: client.TestScannerTimeout#test2772 Thread=293, OpenFileDescriptor=571, MaxFileDescriptor=60000, SystemLoadAverage=115, ProcessCount=165, AvailableMemoryMB=5200 2023-07-14 19:03:30,668 INFO [Listener at localhost.localdomain/40851] hbase.LocalHBaseCluster(262): Not alive RS:1;jenkins-hbase20:42249 2023-07-14 19:03:30,668 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(3275): A region server is stopped or stopping:jenkins-hbase20.apache.org,42249,1689361388520 2023-07-14 19:03:30,668 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(104): START************ test2772 2023-07-14 19:03:30,671 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2627): Found 1 regions for table t 2023-07-14 19:03:30,672 DEBUG [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(2633): firstRegionName=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,672 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(139): Connect 0x5e66a8ae to 127.0.0.1:51669 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-07-14 19:03:30,676 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@12949d6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-07-14 19:03:30,677 ERROR [Listener at localhost.localdomain/40851] helpers.MarkerIgnoringBase(143): ***** ABORTING region server jenkins-hbase20.apache.org,34333,1689361388479: die! ***** 2023-07-14 19:03:30,677 ERROR [Listener at localhost.localdomain/40851] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2023-07-14 19:03:30,677 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for java.lang:type=Memory 2023-07-14 19:03:30,678 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=IPC 2023-07-14 19:03:30,683 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Replication 2023-07-14 19:03:30,685 DEBUG [Listener at localhost.localdomain/40851] util.JSONBean(130): Listing beans for Hadoop:service=HBase,name=RegionServer,sub=Server 2023-07-14 19:03:30,701 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2557): Dump of metrics as JSON on abort: { "beans": [ { "name": "java.lang:type=Memory", "modelerType": "sun.management.MemoryImpl", "ObjectPendingFinalizationCount": 0, "HeapMemoryUsage": { "committed": 1012400128, "init": 524288000, "max": 2051014656, "used": 246819480 }, "NonHeapMemoryUsage": { "committed": 122445824, "init": 2555904, "max": -1, "used": 120078272 }, "Verbose": false, "ObjectName": "java.lang:type=Memory" } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType": "RegionServer,sub=IPC", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "queueSize": 0, "numCallsInGeneralQueue": 0, "numCallsInReplicationQueue": 0, "numCallsInPriorityQueue": 0, "numCallsInMetaPriorityQueue": 0, "numOpenConnections": 3, "numActiveHandler": 0, "numActiveGeneralHandler": 0, "numActivePriorityHandler": 0, "numActiveReplicationHandler": 0, "numGeneralCallsDropped": 0, "numLifoModeSwitches": 0, "numCallsInWriteQueue": 0, "numCallsInReadQueue": 0, "numCallsInScanQueue": 0, "numActiveWriteHandler": 0, "numActiveReadHandler": 0, "numActiveScanHandler": 0, "nettyDirectMemoryUsage": 67108864, "receivedBytes": 11985, "exceptions.RegionMovedException": 0, "authenticationSuccesses": 0, "authorizationFailures": 0, "exceptions.requestTooBig": 0, "exceptions.OutOfOrderScannerNextException": 0, "exceptions.rpcThrottling": 0, "exceptions.otherExceptions": 3, "ProcessCallTime_num_ops": 61, "ProcessCallTime_min": 1, "ProcessCallTime_max": 1, "ProcessCallTime_mean": 1, "ProcessCallTime_25th_percentile": 1, "ProcessCallTime_median": 1, "ProcessCallTime_75th_percentile": 1, "ProcessCallTime_90th_percentile": 1, "ProcessCallTime_95th_percentile": 1, "ProcessCallTime_98th_percentile": 1, "ProcessCallTime_99th_percentile": 1, "ProcessCallTime_99.9th_percentile": 1, "ProcessCallTime_TimeRangeCount_0-1": 2, "exceptions.callQueueTooBig": 0, "QueueCallTime_num_ops": 61, "QueueCallTime_min": 0, "QueueCallTime_max": 0, "QueueCallTime_mean": 0, "QueueCallTime_25th_percentile": 0, "QueueCallTime_median": 0, "QueueCallTime_75th_percentile": 0, "QueueCallTime_90th_percentile": 0, "QueueCallTime_95th_percentile": 0, "QueueCallTime_98th_percentile": 0, "QueueCallTime_99th_percentile": 0, "QueueCallTime_99.9th_percentile": 0, "QueueCallTime_TimeRangeCount_0-1": 2, "authenticationFailures": 0, "exceptions.multiResponseTooLarge": 0, "TotalCallTime_num_ops": 61, "TotalCallTime_min": 1, "TotalCallTime_max": 1, "TotalCallTime_mean": 1, "TotalCallTime_25th_percentile": 1, "TotalCallTime_median": 1, "TotalCallTime_75th_percentile": 1, "TotalCallTime_90th_percentile": 1, "TotalCallTime_95th_percentile": 1, "TotalCallTime_98th_percentile": 1, "TotalCallTime_99th_percentile": 1, "TotalCallTime_99.9th_percentile": 1, "TotalCallTime_TimeRangeCount_0-1": 2, "exceptions.RegionTooBusyException": 0, "exceptions.FailedSanityCheckException": 0, "ResponseSize_num_ops": 61, "ResponseSize_min": 196, "ResponseSize_max": 680, "ResponseSize_mean": 438, "ResponseSize_25th_percentile": 196, "ResponseSize_median": 197, "ResponseSize_75th_percentile": 677, "ResponseSize_90th_percentile": 679, "ResponseSize_95th_percentile": 679, "ResponseSize_98th_percentile": 679, "ResponseSize_99th_percentile": 679, "ResponseSize_99.9th_percentile": 679, "ResponseSize_SizeRangeCount_100-1000": 2, "exceptions.UnknownScannerException": 1, "exceptions": 4, "authenticationFallbacks": 0, "exceptions.quotaExceeded": 0, "exceptions.NotServingRegionException": 0, "authorizationSuccesses": 0, "exceptions.ScannerResetException": 0, "RequestSize_num_ops": 61, "RequestSize_min": 106, "RequestSize_max": 125, "RequestSize_mean": 115, "RequestSize_25th_percentile": 106, "RequestSize_median": 106, "RequestSize_75th_percentile": 124, "RequestSize_90th_percentile": 124, "RequestSize_95th_percentile": 124, "RequestSize_98th_percentile": 124, "RequestSize_99th_percentile": 124, "RequestSize_99.9th_percentile": 124, "RequestSize_SizeRangeCount_100-1000": 2, "sentBytes": 16697 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType": "RegionServer,sub=Replication", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "source.shippedHFiles": 0, "Source.ageOfLastShippedOp_num_ops": 0, "Source.ageOfLastShippedOp_min": 0, "Source.ageOfLastShippedOp_max": 0, "Source.ageOfLastShippedOp_mean": 0, "Source.ageOfLastShippedOp_25th_percentile": 0, "Source.ageOfLastShippedOp_median": 0, "Source.ageOfLastShippedOp_75th_percentile": 0, "Source.ageOfLastShippedOp_90th_percentile": 0, "Source.ageOfLastShippedOp_95th_percentile": 0, "Source.ageOfLastShippedOp_98th_percentile": 0, "Source.ageOfLastShippedOp_99th_percentile": 0, "Source.ageOfLastShippedOp_99.9th_percentile": 0, "source.uncleanlyClosedLogs": 0, "source.closedLogsWithUnknownFileLength": 0, "source.walReaderEditsBufferUsage": 0, "source.repeatedLogFileBytes": 0, "source.shippedKBs": 0, "source.sizeOfHFileRefsQueue": 0, "source.logReadInBytes": 0, "source.completedRecoverQueues": 0, "source.sizeOfLogQueue": 0, "source.restartedLogReading": 0, "source.failedRecoverQueues": 0, "source.ignoredUncleanlyClosedLogContentsInBytes": 0, "Sink.ageOfLastAppliedOp_num_ops": 0, "Sink.ageOfLastAppliedOp_min": 0, "Sink.ageOfLastAppliedOp_max": 0, "Sink.ageOfLastAppliedOp_mean": 0, "Sink.ageOfLastAppliedOp_25th_percentile": 0, "Sink.ageOfLastAppliedOp_median": 0, "Sink.ageOfLastAppliedOp_75th_percentile": 0, "Sink.ageOfLastAppliedOp_90th_percentile": 0, "Sink.ageOfLastAppliedOp_95th_percentile": 0, "Sink.ageOfLastAppliedOp_98th_percentile": 0, "Sink.ageOfLastAppliedOp_99th_percentile": 0, "Sink.ageOfLastAppliedOp_99.9th_percentile": 0, "source.logEditsRead": 0, "source.numInitializing": 0, "source.shippedOps": 0, "sink.appliedHFiles": 0, "source.logEditsFiltered": 0, "source.shippedBytes": 0, "sink.appliedOps": 0, "source.completedLogs": 0, "source.failedBatches": 0, "sink.failedBatches": 0, "source.shippedBatches": 0, "sink.appliedBatches": 0 } ], "beans": [ { "name": "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType": "RegionServer,sub=Server", "tag.zookeeperQuorum": "127.0.0.1:51669", "tag.serverName": "jenkins-hbase20.apache.org,42249,1689361388520", "tag.clusterId": "cfbfac95-a50f-45d8-a4bb-94c79ffe7eeb", "tag.Context": "regionserver", "tag.Hostname": "jenkins-hbase20.apache.org", "regionCount": 0, "storeCount": 0, "hlogFileCount": 1, "hlogFileSize": 0, "storeFileCount": 0, "maxStoreFileCount": 0, "memStoreSize": 0, "storeFileSize": 0, "maxStoreFileAge": 0, "minStoreFileAge": 0, "avgStoreFileAge": 0, "numReferenceFiles": 0, "regionServerStartTime": 1689361388520, "averageRegionSize": 0, "storeFileIndexSize": 0, "staticIndexSize": 0, "staticBloomSize": 0, "mutationsWithoutWALCount": 0, "mutationsWithoutWALSize": 0, "percentFilesLocal": 0.0, "percentFilesLocalSecondaryRegions": 0.0, "totalBytesRead": 546, "localBytesRead": 546, "shortCircuitBytesRead": 0, "zeroCopyBytesRead": 0, "splitQueueLength": 0, "compactionQueueLength": 0, "smallCompactionQueueLength": 0, "largeCompactionQueueLength": 0, "flushQueueLength": 0, "blockCacheFreeSize": 819803968, "blockCacheCount": 0, "blockCacheDataBlockCount": 0, "blockCacheSize": 601920, "blockCacheCountHitPercent": 0.0, "blockCacheExpressHitPercent": 0.0, "l1CacheSize": 601920, "l1CacheFreeSize": 819803968, "l1CacheCount": 0, "l1CacheEvictionCount": 0, "l1CacheHitCount": 0, "l1CacheMissCount": 0, "l1CacheHitRatio": 0.0, "l1CacheMissRatio": 0.0, "l2CacheSize": 0, "l2CacheFreeSize": 0, "l2CacheCount": 0, "l2CacheEvictionCount": 0, "l2CacheHitCount": 0, "l2CacheMissCount": 0, "l2CacheHitRatio": 0.0, "l2CacheMissRatio": 0.0, "mobFileCacheCount": 0, "mobFileCacheHitPercent": 0.0, "readRequestRatePerSecond": 0.0, "writeRequestRatePerSecond": 0.0, "ByteBuffAllocatorHeapAllocationBytes": 1640, "ByteBuffAllocatorPoolAllocationBytes": 66560, "ByteBuffAllocatorHeapAllocationRatio": 0.0, "ByteBuffAllocatorTotalBufferCount": 186, "ByteBuffAllocatorUsedBufferCount": 1, "totalRequestCount": 11, "totalRowActionRequestCount": 0, "readRequestCount": 0, "filteredReadRequestCount": 0, "writeRequestCount": 0, "rpcGetRequestCount": 0, "rpcFullScanRequestCount": 1, "rpcScanRequestCount": 1, "rpcMultiRequestCount": 0, "rpcMutateRequestCount": 10, "checkMutateFailedCount": 0, "checkMutatePassedCount": 0, "blockCacheHitCount": 0, "blockCacheHitCountPrimary": 0, "blockCacheHitCachingCount": 0, "blockCacheMissCount": 0, "blockCacheMissCountPrimary": 0, "blockCacheMissCachingCount": 0, "blockCacheEvictionCount": 0, "blockCacheEvictionCountPrimary": 0, "blockCacheFailedInsertionCount": 0, "blockCacheDataMissCount": 0, "blockCacheLeafIndexMissCount": 0, "blockCacheBloomChunkMissCount": 0, "blockCacheMetaMissCount": 0, "blockCacheRootIndexMissCount": 0, "blockCacheIntermediateIndexMissCount": 0, "blockCacheFileInfoMissCount": 0, "blockCacheGeneralBloomMetaMissCount": 0, "blockCacheDeleteFamilyBloomMissCount": 0, "blockCacheTrailerMissCount": 0, "blockCacheDataHitCount": 0, "blockCacheLeafIndexHitCount": 0, "blockCacheBloomChunkHitCount": 0, "blockCacheMetaHitCount": 0, "blockCacheRootIndexHitCount": 0, "blockCacheIntermediateIndexHitCount": 0, "blockCacheFileInfoHitCount": 0, "blockCacheGeneralBloomMetaHitCount": 0, "blockCacheDeleteFamilyBloomHitCount": 0, "blockCacheTrailerHitCount": 0, "updatesBlockedTime": 0, "flushedCellsCount": 0, "compactedCellsCount": 0, "majorCompactedCellsCount": 0, "flushedCellsSize": 0, "compactedCellsSize": 0, "majorCompactedCellsSize": 0, "cellsCountCompactedFromMob": 0, "cellsCountCompactedToMob": 0, "cellsSizeCompactedFromMob": 0, "cellsSizeCompactedToMob": 0, "mobFlushCount": 0, "mobFlushedCellsCount": 0, "mobFlushedCellsSize": 0, "mobScanCellsCount": 0, "mobScanCellsSize": 0, "mobFileCacheAccessCount": 0, "mobFileCacheMissCount": 0, "mobFileCacheEvictedCount": 0, "hedgedReads": 0, "hedgedReadWins": 0, "hedgedReadOpsInCurThread": 0, "blockedRequestCount": 0, "CheckAndMutate_num_ops": 0, "CheckAndMutate_min": 0, "CheckAndMutate_max": 0, "CheckAndMutate_mean": 0, "CheckAndMutate_25th_percentile": 0, "CheckAndMutate_median": 0, "CheckAndMutate_75th_percentile": 0, "CheckAndMutate_90th_percentile": 0, "CheckAndMutate_95th_percentile": 0, "CheckAndMutate_98th_percentile": 0, "CheckAndMutate_99th_percentile": 0, "CheckAndMutate_99.9th_percentile": 0, "MajorCompactionTime_num_ops": 0, "MajorCompactionTime_min": 0, "MajorCompactionTime_max": 0, "MajorCompactionTime_mean": 0, "MajorCompactionTime_25th_percentile": 0, "MajorCompactionTime_median": 0, "MajorCompactionTime_75th_percentile": 0, "MajorCompactionTime_90th_percentile": 0, "MajorCompactionTime_95th_percentile": 0, "MajorCompactionTime_98th_percentile": 0, "MajorCompactionTime_99th_percentile": 0, "MajorCompactionTime_99.9th_percentile": 0, "PauseTimeWithGc_num_ops": 0, "PauseTimeWithGc_min": 0, "PauseTimeWithGc_max": 0, "PauseTimeWithGc_mean": 0, "PauseTimeWithGc_25th_percentile": 0, "PauseTimeWithGc_median": 0, "PauseTimeWithGc_75th_percentile": 0, "PauseTimeWithGc_90th_percentile": 0, "PauseTimeWithGc_95th_percentile": 0, "PauseTimeWithGc_98th_percentile": 0, "PauseTimeWithGc_99th_percentile": 0, "PauseTimeWithGc_99.9th_percentile": 0, "compactedOutputBytes": 0, "pauseWarnThresholdExceeded": 0, "ScanTime_num_ops": 1, "ScanTime_min": 0, "ScanTime_max": 0, "ScanTime_mean": 0, "ScanTime_25th_percentile": 0, "ScanTime_median": 0, "ScanTime_75th_percentile": 0, "ScanTime_90th_percentile": 0, "ScanTime_95th_percentile": 0, "ScanTime_98th_percentile": 0, "ScanTime_99th_percentile": 0, "ScanTime_99.9th_percentile": 0, "Increment_num_ops": 0, "Increment_min": 0, "Increment_max": 0, "Increment_mean": 0, "Increment_25th_percentile": 0, "Increment_median": 0, "Increment_75th_percentile": 0, "Increment_90th_percentile": 0, "Increment_95th_percentile": 0, "Increment_98th_percentile": 0, "Increment_99th_percentile": 0, "Increment_99.9th_percentile": 0, "Delete_num_ops": 0, "Delete_min": 0, "Delete_max": 0, "Delete_mean": 0, "Delete_25th_percentile": 0, "Delete_median": 0, "Delete_75th_percentile": 0, "Delete_90th_percentile": 0, "Delete_95th_percentile": 0, "Delete_98th_percentile": 0, "Delete_99th_percentile": 0, "Delete_99.9th_percentile": 0, "Put_num_ops": 10, "Put_min": 0, "Put_max": 0, "Put_mean": 0, "Put_25th_percentile": 0, "Put_median": 0, "Put_75th_percentile": 0, "Put_90th_percentile": 0, "Put_95th_percentile": 0, "Put_98th_percentile": 0, "Put_99th_percentile": 0, "Put_99.9th_percentile": 0, "DeleteBatch_num_ops": 0, "DeleteBatch_min": 0, "DeleteBatch_max": 0, "DeleteBatch_mean": 0, "DeleteBatch_25th_percentile": 0, "DeleteBatch_median": 0, "DeleteBatch_75th_percentile": 0, "DeleteBatch_90th_percentile": 0, "DeleteBatch_95th_percentile": 0, "DeleteBatch_98th_percentile": 0, "DeleteBatch_99th_percentile": 0, "DeleteBatch_99.9th_percentile": 0, "splitRequestCount": 0, "FlushMemstoreSize_num_ops": 0, "FlushMemstoreSize_min": 0, "FlushMemstoreSize_max": 0, "FlushMemstoreSize_mean": 0, "FlushMemstoreSize_25th_percentile": 0, "FlushMemstoreSize_median": 0, "FlushMemstoreSize_75th_percentile": 0, "FlushMemstoreSize_90th_percentile": 0, "FlushMemstoreSize_95th_percentile": 0, "FlushMemstoreSize_98th_percentile": 0, "FlushMemstoreSize_99th_percentile": 0, "FlushMemstoreSize_99.9th_percentile": 0, "CompactionInputFileCount_num_ops": 0, "CompactionInputFileCount_min": 0, "CompactionInputFileCount_max": 0, "CompactionInputFileCount_mean": 0, "CompactionInputFileCount_25th_percentile": 0, "CompactionInputFileCount_median": 0, "CompactionInputFileCount_75th_percentile": 0, "CompactionInputFileCount_90th_percentile": 0, "CompactionInputFileCount_95th_percentile": 0, "CompactionInputFileCount_98th_percentile": 0, "CompactionInputFileCount_99th_percentile": 0, "CompactionInputFileCount_99.9th_percentile": 0, "PutBatch_num_ops": 0, "PutBatch_min": 0, "PutBatch_max": 0, "PutBatch_mean": 0, "PutBatch_25th_percentile": 0, "PutBatch_median": 0, "PutBatch_75th_percentile": 0, "PutBatch_90th_percentile": 0, "PutBatch_95th_percentile": 0, "PutBatch_98th_percentile": 0, "PutBatch_99th_percentile": 0, "PutBatch_99.9th_percentile": 0, "CompactionTime_num_ops": 0, "CompactionTime_min": 0, "CompactionTime_max": 0, "CompactionTime_mean": 0, "CompactionTime_25th_percentile": 0, "CompactionTime_median": 0, "CompactionTime_75th_percentile": 0, "CompactionTime_90th_percentile": 0, "CompactionTime_95th_percentile": 0, "CompactionTime_98th_percentile": 0, "CompactionTime_99th_percentile": 0, "CompactionTime_99.9th_percentile": 0, "Get_num_ops": 0, "Get_min": 0, "Get_max": 0, "Get_mean": 0, "Get_25th_percentile": 0, "Get_median": 0, "Get_75th_percentile": 0, "Get_90th_percentile": 0, "Get_95th_percentile": 0, "Get_98th_percentile": 0, "Get_99th_percentile": 0, "Get_99.9th_percentile": 0, "MajorCompactionInputFileCount_num_ops": 0, "MajorCompactionInputFileCount_min": 0, "MajorCompactionInputFileCount_max": 0, "MajorCompactionInputFileCount_mean": 0, "MajorCompactionInputFileCount_25th_percentile": 0, "MajorCompactionInputFileCount_median": 0, "MajorCompactionInputFileCount_75th_percentile": 0, "MajorCompactionInputFileCount_90th_percentile": 0, "MajorCompactionInputFileCount_95th_percentile": 0, "MajorCompactionInputFileCount_98th_percentile": 0, "MajorCompactionInputFileCount_99th_percentile": 0, "MajorCompactionInputFileCount_99.9th_percentile": 0, "CheckAndPut_num_ops": 0, "CheckAndPut_min": 0, "CheckAndPut_max": 0, "CheckAndPut_mean": 0, "CheckAndPut_25th_percentile": 0, "CheckAndPut_median": 0, "CheckAndPut_75th_percentile": 0, "CheckAndPut_90th_percentile": 0, "CheckAndPut_95th_percentile": 0, "CheckAndPut_98th_percentile": 0, "CheckAndPut_99th_percentile": 0, "CheckAndPut_99.9th_percentile": 0, "SplitTime_num_ops": 0, "SplitTime_min": 0, "SplitTime_max": 0, "SplitTime_mean": 0, "SplitTime_25th_percentile": 0, "SplitTime_median": 0, "SplitTime_75th_percentile": 0, "SplitTime_90th_percentile": 0, "SplitTime_95th_percentile": 0, "SplitTime_98th_percentile": 0, "SplitTime_99th_percentile": 0, "SplitTime_99.9th_percentile": 0, "MajorCompactionOutputSize_num_ops": 0, "MajorCompactionOutputSize_min": 0, "MajorCompactionOutputSize_max": 0, "MajorCompactionOutputSize_mean": 0, "MajorCompactionOutputSize_25th_percentile": 0, "MajorCompactionOutputSize_median": 0, "MajorCompactionOutputSize_75th_percentile": 0, "MajorCompactionOutputSize_90th_percentile": 0, "MajorCompactionOutputSize_95th_percentile": 0, "MajorCompactionOutputSize_98th_percentile": 0, "MajorCompactionOutputSize_99th_percentile": 0, "MajorCompactionOutputSize_99.9th_percentile": 0, "majorCompactedInputBytes": 0, "slowAppendCount": 0, "flushedOutputBytes": 0, "CompactionOutputFileCount_num_ops": 0, "CompactionOutputFileCount_min": 0, "CompactionOutputFileCount_max": 0, "CompactionOutputFileCount_mean": 0, "CompactionOutputFileCount_25th_percentile": 0, "CompactionOutputFileCount_median": 0, "CompactionOutputFileCount_75th_percentile": 0, "CompactionOutputFileCount_90th_percentile": 0, "CompactionOutputFileCount_95th_percentile": 0, "CompactionOutputFileCount_98th_percentile": 0, "CompactionOutputFileCount_99th_percentile": 0, "CompactionOutputFileCount_99.9th_percentile": 0, "slowDeleteCount": 0, "Replay_num_ops": 0, "Replay_min": 0, "Replay_max": 0, "Replay_mean": 0, "Replay_25th_percentile": 0, "Replay_median": 0, "Replay_75th_percentile": 0, "Replay_90th_percentile": 0, "Replay_95th_percentile": 0, "Replay_98th_percentile": 0, "Replay_99th_percentile": 0, "Replay_99.9th_percentile": 0, "FlushTime_num_ops": 0, "FlushTime_min": 0, "FlushTime_max": 0, "FlushTime_mean": 0, "FlushTime_25th_percentile": 0, "FlushTime_median": 0, "FlushTime_75th_percentile": 0, "FlushTime_90th_percentile": 0, "FlushTime_95th_percentile": 0, "FlushTime_98th_percentile": 0, "FlushTime_99th_percentile": 0, "FlushTime_99.9th_percentile": 0, "MajorCompactionInputSize_num_ops": 0, "MajorCompactionInputSize_min": 0, "MajorCompactionInputSize_max": 0, "MajorCompactionInputSize_mean": 0, "MajorCompactionInputSize_25th_percentile": 0, "MajorCompactionInputSize_median": 0, "MajorCompactionInputSize_75th_percentile": 0, "MajorCompactionInputSize_90th_percentile": 0, "MajorCompactionInputSize_95th_percentile": 0, "MajorCompactionInputSize_98th_percentile": 0, "MajorCompactionInputSize_99th_percentile": 0, "MajorCompactionInputSize_99.9th_percentile": 0, "pauseInfoThresholdExceeded": 0, "splitSuccessCount": 0, "CheckAndDelete_num_ops": 0, "CheckAndDelete_min": 0, "CheckAndDelete_max": 0, "CheckAndDelete_mean": 0, "CheckAndDelete_25th_percentile": 0, "CheckAndDelete_median": 0, "CheckAndDelete_75th_percentile": 0, "CheckAndDelete_90th_percentile": 0, "CheckAndDelete_95th_percentile": 0, "CheckAndDelete_98th_percentile": 0, "CheckAndDelete_99th_percentile": 0, "CheckAndDelete_99.9th_percentile": 0, "CompactionInputSize_num_ops": 0, "CompactionInputSize_min": 0, "CompactionInputSize_max": 0, "CompactionInputSize_mean": 0, "CompactionInputSize_25th_percentile": 0, "CompactionInputSize_median": 0, "CompactionInputSize_75th_percentile": 0, "CompactionInputSize_90th_percentile": 0, "CompactionInputSize_95th_percentile": 0, "CompactionInputSize_98th_percentile": 0, "CompactionInputSize_99th_percentile": 0, "CompactionInputSize_99.9th_percentile": 0, "MajorCompactionOutputFileCount_num_ops": 0, "MajorCompactionOutputFileCount_min": 0, "MajorCompactionOutputFileCount_max": 0, "MajorCompactionOutputFileCount_mean": 0, "MajorCompactionOutputFileCount_25th_percentile": 0, "MajorCompactionOutputFileCount_median": 0, "MajorCompactionOutputFileCount_75th_percentile": 0, "MajorCompactionOutputFileCount_90th_percentile": 0, "MajorCompactionOutputFileCount_95th_percentile": 0, "MajorCompactionOutputFileCount_98th_percentile": 0, "MajorCompactionOutputFileCount_99th_percentile": 0, "MajorCompactionOutputFileCount_99.9th_percentile": 0, "ScanSize_num_ops": 1, "ScanSize_min": 0, "ScanSize_max": 0, "ScanSize_mean": 0, "ScanSize_25th_percentile": 0, "ScanSize_median": 0, "ScanSize_75th_percentile": 0, "ScanSize_90th_percentile": 0, "ScanSize_95th_percentile": 0, "ScanSize_98th_percentile": 0, "ScanSize_99th_percentile": 0, "ScanSize_99.9th_percentile": 0, "slowGetCount": 0, "flushedMemstoreBytes": 0, "CompactionOutputSize_num_ops": 0, "CompactionOutputSize_min": 0, "CompactionOutputSize_max": 0, "CompactionOutputSize_mean": 0, "CompactionOutputSize_25th_percentile": 0, "CompactionOutputSize_median": 0, "CompactionOutputSize_75th_percentile": 0, "CompactionOutputSize_90th_percentile": 0, "CompactionOutputSize_95th_percentile": 0, "CompactionOutputSize_98th_percentile": 0, "CompactionOutputSize_99th_percentile": 0, "CompactionOutputSize_99.9th_percentile": 0, "majorCompactedOutputBytes": 0, "PauseTimeWithoutGc_num_ops": 0, "PauseTimeWithoutGc_min": 0, "PauseTimeWithoutGc_max": 0, "PauseTimeWithoutGc_mean": 0, "PauseTimeWithoutGc_25th_percentile": 0, "PauseTimeWithoutGc_median": 0, "PauseTimeWithoutGc_75th_percentile": 0, "PauseTimeWithoutGc_90th_percentile": 0, "PauseTimeWithoutGc_95th_percentile": 0, "PauseTimeWithoutGc_98th_percentile": 0, "PauseTimeWithoutGc_99th_percentile": 0, "PauseTimeWithoutGc_99.9th_percentile": 0, "slowPutCount": 0, "slowIncrementCount": 0, "compactedInputBytes": 0, "Append_num_ops": 0, "Append_min": 0, "Append_max": 0, "Append_mean": 0, "Append_25th_percentile": 0, "Append_median": 0, "Append_75th_percentile": 0, "Append_90th_percentile": 0, "Append_95th_percentile": 0, "Append_98th_percentile": 0, "Append_99th_percentile": 0, "Append_99.9th_percentile": 0, "FlushOutputSize_num_ops": 0, "FlushOutputSize_min": 0, "FlushOutputSize_max": 0, "FlushOutputSize_mean": 0, "FlushOutputSize_25th_percentile": 0, "FlushOutputSize_median": 0, "FlushOutputSize_75th_percentile": 0, "FlushOutputSize_90th_percentile": 0, "FlushOutputSize_95th_percentile": 0, "FlushOutputSize_98th_percentile": 0, "FlushOutputSize_99th_percentile": 0, "FlushOutputSize_99.9th_percentile": 0, "Bulkload_count": 0, "Bulkload_mean_rate": 0.0, "Bulkload_1min_rate": 0.0, "Bulkload_5min_rate": 0.0, "Bulkload_15min_rate": 0.0, "Bulkload_num_ops": 0, "Bulkload_min": 0, "Bulkload_max": 0, "Bulkload_mean": 0, "Bulkload_25th_percentile": 0, "Bulkload_median": 0, "Bulkload_75th_percentile": 0, "Bulkload_90th_percentile": 0, "Bulkload_95th_percentile": 0, "Bulkload_98th_percentile": 0, "Bulkload_99th_percentile": 0, "Bulkload_99.9th_percentile": 0, "ServerReadQueryPerSecond_count": 54, "ServerReadQueryPerSecond_mean_rate": 2.556075659024971, "ServerReadQueryPerSecond_1min_rate": 7.476487517485488, "ServerReadQueryPerSecond_5min_rate": 9.131802475206852, "ServerReadQueryPerSecond_15min_rate": 9.441325956687528, "ServerWriteQueryPerSecond_count": 24, "ServerWriteQueryPerSecond_mean_rate": 1.136033529978137, "ServerWriteQueryPerSecond_1min_rate": 3.738243758742744, "ServerWriteQueryPerSecond_5min_rate": 4.565901237603426, "ServerWriteQueryPerSecond_15min_rate": 4.720662978343764 } ] } 2023-07-14 19:03:30,702 WARN [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38349] master.MasterRpcServices(612): jenkins-hbase20.apache.org,34333,1689361388479 reported a fatal error: ***** ABORTING region server jenkins-hbase20.apache.org,34333,1689361388479: die! ***** 2023-07-14 19:03:30,704 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,34333,1689361388479' ***** 2023-07-14 19:03:30,705 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2311): STOPPED: die! 2023-07-14 19:03:30,705 INFO [RS:0;jenkins-hbase20:34333] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 19:03:30,706 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 19:03:30,706 INFO [RS:0;jenkins-hbase20:34333] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager abruptly. 2023-07-14 19:03:30,706 INFO [RS:0;jenkins-hbase20:34333] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager abruptly. 2023-07-14 19:03:30,706 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(3305): Received CLOSE for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:30,707 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(3305): Received CLOSE for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:30,707 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1141): aborting server jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:30,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c5039524108f6f5f0d888c4d54df6bc4, disabling compactions & flushes 2023-07-14 19:03:30,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,707 DEBUG [RS:0;jenkins-hbase20:34333] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1569481a to 127.0.0.1:51669 2023-07-14 19:03:30,707 DEBUG [RS:0;jenkins-hbase20:34333] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:30,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,707 INFO [RS:0;jenkins-hbase20:34333] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 19:03:30,708 INFO [RS:0;jenkins-hbase20:34333] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 19:03:30,707 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. after waiting 0 ms 2023-07-14 19:03:30,708 INFO [RS:0;jenkins-hbase20:34333] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 19:03:30,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,708 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 19:03:30,714 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 19:03:30,714 ERROR [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 78 in region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,714 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 19:03:30,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,715 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 19:03:30,714 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1478): Online Regions={c5039524108f6f5f0d888c4d54df6bc4=hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4., f8ff06c91c03cb8152ab689acbbf13c3=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3., 1588230740=hbase:meta,,1.1588230740} 2023-07-14 19:03:30,716 DEBUG [hconnection-0x75745710-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:30,716 DEBUG [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1504): Waiting on 1588230740, c5039524108f6f5f0d888c4d54df6bc4, f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:30,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 19:03:30,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c5039524108f6f5f0d888c4d54df6bc4: 2023-07-14 19:03:30,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 19:03:30,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 19:03:30,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:30,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f8ff06c91c03cb8152ab689acbbf13c3, disabling compactions & flushes 2023-07-14 19:03:30,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,717 ERROR [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1825): Memstore data size is 3234 in region hbase:meta,,1.1588230740 2023-07-14 19:03:30,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. after waiting 0 ms 2023-07-14 19:03:30,717 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 19:03:30,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,719 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 19:03:30,720 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 19:03:30,720 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 19:03:30,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:30,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:30,722 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43688, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:30,722 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34333] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,34333,1689361388479 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:30,722 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34333] ipc.CallRunner(144): callId: 0 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:43688 deadline: 1689361470722, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,34333,1689361388479 aborting 2023-07-14 19:03:30,826 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34333] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,34333,1689361388479 aborting at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1524) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3494) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-07-14 19:03:30,826 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34333] ipc.CallRunner(144): callId: 1 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:43688 deadline: 1689361470826, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server jenkins-hbase20.apache.org,34333,1689361388479 aborting 2023-07-14 19:03:30,916 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,34333,1689361388479; all regions closed. 2023-07-14 19:03:30,930 DEBUG [RS:0;jenkins-hbase20:34333] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:30,930 INFO [RS:0;jenkins-hbase20:34333] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:30,930 INFO [RS:0;jenkins-hbase20:34333] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 19:03:30,931 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 19:03:30,931 INFO [RS:0;jenkins-hbase20:34333] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:34333 2023-07-14 19:03:30,934 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:30,934 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:30,934 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 2023-07-14 19:03:30,934 ERROR [Listener at localhost.localdomain/40851-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3147eceb rejected from java.util.concurrent.ThreadPoolExecutor@35786e1b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:603) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-07-14 19:03:30,934 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:30,935 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:30,935 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,34333,1689361388479] 2023-07-14 19:03:30,935 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,34333,1689361388479; numProcessing=1 2023-07-14 19:03:30,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:30,936 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase20.apache.org,34333,1689361388479 znode expired, triggering replicatorRemoved event 2023-07-14 19:03:30,936 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,34333,1689361388479 already deleted, retry=false 2023-07-14 19:03:30,936 INFO [RegionServerTracker-0] master.ServerManager(568): Processing expiration of jenkins-hbase20.apache.org,34333,1689361388479 on jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:30,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:30,938 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1029): Stored pid=17, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure jenkins-hbase20.apache.org,34333,1689361388479, splitWal=true, meta=true 2023-07-14 19:03:30,938 INFO [RegionServerTracker-0] assignment.AssignmentManager(1734): Scheduled ServerCrashProcedure pid=17 for jenkins-hbase20.apache.org,34333,1689361388479 (carryingMeta=true) jenkins-hbase20.apache.org,34333,1689361388479/CRASHED/regionCount=3/lock=java.util.concurrent.locks.ReentrantReadWriteLock@3888a39a[Write locks = 1, Read locks = 0], oldState=ONLINE. 2023-07-14 19:03:30,940 INFO [PEWorker-2] procedure.ServerCrashProcedure(161): Start pid=17, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,34333,1689361388479, splitWal=true, meta=true 2023-07-14 19:03:30,942 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,34333,1689361388479, splitWal=true, meta=true, isMeta: true 2023-07-14 19:03:30,946 DEBUG [PEWorker-2] master.MasterWalManager(318): Renamed region directory: hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting 2023-07-14 19:03:30,948 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,34333,1689361388479 WAL count=1, meta=true 2023-07-14 19:03:30,949 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta}] 2023-07-14 19:03:30,952 DEBUG [PEWorker-1] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:30,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=18, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, worker=jenkins-hbase20.apache.org,38019,1689361393499}] 2023-07-14 19:03:31,029 WARN [RPCClient-NioEventLoopGroup-5-7] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:34333 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:34333 Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:31,030 DEBUG [RPCClient-NioEventLoopGroup-5-7] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:34333 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:34333 2023-07-14 19:03:31,035 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:31,035 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:34333-0x1008cdd89820001, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:31,035 INFO [RS:0;jenkins-hbase20:34333] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,34333,1689361388479; zookeeper connection closed. 2023-07-14 19:03:31,036 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@d803c3e] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@d803c3e 2023-07-14 19:03:31,106 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,106 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-07-14 19:03:31,108 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43340, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-07-14 19:03:31,108 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=38019] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=19 2023-07-14 19:03:31,126 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, size=4.0 K (4130bytes) 2023-07-14 19:03:31,126 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta 2023-07-14 19:03:31,127 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta after 1ms 2023-07-14 19:03:31,131 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(311): Open hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta took 5ms 2023-07-14 19:03:31,136 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(346): Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2023-07-14 19:03:31,138 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta so closing down 2023-07-14 19:03:31,138 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 19:03:31,142 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp 2023-07-14 19:03:31,142 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 19:03:31,155 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) 2023-07-14 19:03:31,157 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 19:03:31,157 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(423): Processed 13 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false 2023-07-14 19:03:31,158 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, journal: Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, size=4.0 K (4130bytes) at 1689361411126Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta so closing down at 1689361411138 (+12 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp at 1689361411142 (+4 ms)3 split writer threads finished at 1689361411142Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp (wrote 12 edits, skipped 0 edits in 0 ms) at 1689361411155 (+13 ms)Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 at 1689361411157 (+2 ms)Processed 13 edits across 1 Regions in 25 ms; skipped=1; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, size=4.0 K, length=4130, corrupted=false, cancelled=false at 1689361411157 2023-07-14 19:03:31,158 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta 2023-07-14 19:03:31,158 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-0] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=19 2023-07-14 19:03:31,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.HMaster(3641): Remote procedure done, pid=19 2023-07-14 19:03:31,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs 2023-07-14 19:03:31,165 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=18 2023-07-14 19:03:31,165 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=18, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, worker=jenkins-hbase20.apache.org,38019,1689361393499 in 210 msec 2023-07-14 19:03:31,167 DEBUG [PEWorker-5] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-07-14 19:03:31,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.meta.1689361390387.meta, worker=jenkins-hbase20.apache.org,38019,1689361393499 in 219 msec 2023-07-14 19:03:31,172 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting dir is empty, no logs to split. 2023-07-14 19:03:31,172 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,34333,1689361388479 WAL count=0, meta=true 2023-07-14 19:03:31,172 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,34333,1689361388479 WAL splitting is done? wals=0, meta=true 2023-07-14 19:03:31,173 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-07-14 19:03:31,174 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-07-14 19:03:31,174 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=20, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 19:03:31,325 DEBUG [jenkins-hbase20:38349] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 19:03:31,325 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38019,1689361393499, state=OPENING 2023-07-14 19:03:31,327 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 19:03:31,327 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38019,1689361393499}] 2023-07-14 19:03:31,327 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 19:03:31,333 WARN [ReadOnlyZKClient-127.0.0.1:51669@0x5e66a8ae] client.ZKConnectionRegistry(168): Meta region is in state OPENING 2023-07-14 19:03:31,333 DEBUG [hconnection-0x75745710-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:31,337 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43344, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:31,338 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38019] ipc.CallRunner(144): callId: 3 service: ClientService methodName: Scan size: 111 connection: 148.251.75.209:43344 deadline: 1689361471338, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:meta,,1.1588230740 2023-07-14 19:03:31,490 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-07-14 19:03:31,493 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C38019%2C1689361393499.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,38019,1689361393499, archiveDir=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs, maxLogs=32 2023-07-14 19:03:31,511 DEBUG [RS-EventLoopGroup-6-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK] 2023-07-14 19:03:31,511 DEBUG [RS-EventLoopGroup-6-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK] 2023-07-14 19:03:31,513 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,38019,1689361393499/jenkins-hbase20.apache.org%2C38019%2C1689361393499.meta.1689361411494.meta 2023-07-14 19:03:31,513 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36773,DS-cf27f661-e167-4e69-8b53-266bf6d793d8,DISK], DatanodeInfoWithStorage[127.0.0.1:42661,DS-d3937e71-9e9e-4434-9e6c-a1eac471555a,DISK]] 2023-07-14 19:03:31,514 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:31,514 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-07-14 19:03:31,514 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-07-14 19:03:31,515 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-07-14 19:03:31,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-07-14 19:03:31,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:31,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-07-14 19:03:31,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-07-14 19:03:31,518 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-07-14 19:03:31,520 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info 2023-07-14 19:03:31,520 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info 2023-07-14 19:03:31,520 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-07-14 19:03:31,521 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:31,522 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-07-14 19:03:31,523 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:31,524 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/rep_barrier 2023-07-14 19:03:31,524 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/rep_barrier 2023-07-14 19:03:31,524 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-07-14 19:03:31,525 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:31,525 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-07-14 19:03:31,527 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table 2023-07-14 19:03:31,527 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table 2023-07-14 19:03:31,528 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-07-14 19:03:31,529 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:31,530 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:31,533 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740 2023-07-14 19:03:31,534 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 19:03:31,541 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 32, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=15, path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 19:03:31,541 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=3.16 KB heapSize=6.27 KB 2023-07-14 19:03:31,558 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=3.00 KB at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/info/af2e22b3b3fd462cac9c234027136f36 2023-07-14 19:03:31,583 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=160 B at sequenceid=15 (bloomFilter=false), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/table/e30384523b2249bead82fb4a5ebf64bf 2023-07-14 19:03:31,592 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/info/af2e22b3b3fd462cac9c234027136f36 as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info/af2e22b3b3fd462cac9c234027136f36 2023-07-14 19:03:31,599 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info/af2e22b3b3fd462cac9c234027136f36, entries=28, sequenceid=15, filesize=7.7 K 2023-07-14 19:03:31,601 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/table/e30384523b2249bead82fb4a5ebf64bf as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table/e30384523b2249bead82fb4a5ebf64bf 2023-07-14 19:03:31,612 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/table/e30384523b2249bead82fb4a5ebf64bf, entries=4, sequenceid=15, filesize=4.7 K 2023-07-14 19:03:31,613 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~3.16 KB/3234, heapSize ~5.98 KB/6128, currentSize=0 B/0 for 1588230740 in 72ms, sequenceid=15, compaction requested=false; wal=null 2023-07-14 19:03:31,614 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/0000000000000000015 2023-07-14 19:03:31,618 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-07-14 19:03:31,620 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-07-14 19:03:31,623 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=1 2023-07-14 19:03:31,625 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70604770, jitterRate=0.052093058824539185}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-07-14 19:03:31,625 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-07-14 19:03:31,626 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:meta,,1.1588230740, pid=21, masterSystemTime=1689361411482 2023-07-14 19:03:31,628 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:meta,,1.1588230740 2023-07-14 19:03:31,629 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:meta,,1.1588230740 2023-07-14 19:03:31,629 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,38019,1689361393499, state=OPEN 2023-07-14 19:03:31,630 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-07-14 19:03:31,630 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-07-14 19:03:31,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-07-14 19:03:31,633 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,38019,1689361393499 in 303 msec 2023-07-14 19:03:31,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=17 2023-07-14 19:03:31,635 INFO [PEWorker-2] procedure.ServerCrashProcedure(199): jenkins-hbase20.apache.org,34333,1689361388479 had 3 regions 2023-07-14 19:03:31,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 460 msec 2023-07-14 19:03:31,636 INFO [PEWorker-2] procedure.ServerCrashProcedure(300): Splitting WALs pid=17, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure jenkins-hbase20.apache.org,34333,1689361388479, splitWal=true, meta=true, isMeta: false 2023-07-14 19:03:31,639 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,34333,1689361388479 WAL count=1, meta=false 2023-07-14 19:03:31,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=17, state=RUNNABLE:ACQUIRE_SPLIT_WAL_WORKER; SplitWALProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789}] 2023-07-14 19:03:31,641 DEBUG [PEWorker-3] master.SplitWALManager(159): Acquired split WAL worker=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,643 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, worker=jenkins-hbase20.apache.org,38019,1689361393499}] 2023-07-14 19:03:31,796 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=38019] regionserver.RSRpcServices(3879): Executing remote procedure class org.apache.hadoop.hbase.regionserver.SplitWALCallable, pid=23 2023-07-14 19:03:31,814 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(300): Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, size=901 (901bytes) 2023-07-14 19:03:31,814 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(86): Recover lease on dfs file hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 2023-07-14 19:03:31,815 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] util.RecoverLeaseFSUtils(175): Recovered lease, attempt=0 on file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 after 1ms 2023-07-14 19:03:31,819 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(311): Open hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 took 5ms 2023-07-14 19:03:31,826 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for c5039524108f6f5f0d888c4d54df6bc4: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2023-07-14 19:03:31,829 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(346): Last flushed sequenceid for f8ff06c91c03cb8152ab689acbbf13c3: last_flushed_sequence_id: 13 store_sequence_id { family_name: "f" sequence_id: 13 } 2023-07-14 19:03:31,829 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(400): Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 so closing down 2023-07-14 19:03:31,829 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(121): Waiting for split writer threads to finish 2023-07-14 19:03:31,835 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1-Writer-2] wal.AbstractRecoveredEditsOutputSink(71): Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp 2023-07-14 19:03:31,835 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.OutputSink(146): 3 split writer threads finished 2023-07-14 19:03:31,855 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(90): Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp (wrote 2 edits, skipped 0 edits in 0 ms) 2023-07-14 19:03:31,857 INFO [split-log-closeStream-pool-0] wal.AbstractRecoveredEditsOutputSink(123): Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000005 2023-07-14 19:03:31,857 INFO [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(423): Processed 4 edits across 1 Regions in 35 ms; skipped=2; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, size=901, length=901, corrupted=false, cancelled=false 2023-07-14 19:03:31,857 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] wal.WALSplitter(426): Completed split of hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, journal: Splitting hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, size=901 (901bytes) at 1689361411814Finishing writing output for hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 so closing down at 1689361411829 (+15 ms)Creating recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp at 1689361411835 (+6 ms)3 split writer threads finished at 1689361411835Closed recovered edits writer path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp (wrote 2 edits, skipped 0 edits in 0 ms) at 1689361411855 (+20 ms)Rename recovered edits hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000004-jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789.temp to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000005 at 1689361411857 (+2 ms)Processed 4 edits across 1 Regions in 35 ms; skipped=2; WAL=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, size=901, length=901, corrupted=false, cancelled=false at 1689361411857 2023-07-14 19:03:31,857 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.SplitLogWorker(136): Done splitting WAL hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 2023-07-14 19:03:31,857 DEBUG [RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0-1] regionserver.RemoteProcedureResultReporter(64): Successfully complete execution of pid=23 2023-07-14 19:03:31,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] master.HMaster(3641): Remote procedure done, pid=23 2023-07-14 19:03:31,861 DEBUG [hconnection-0x75745710-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:34333 this server is in the failed servers list 2023-07-14 19:03:31,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38349] wal.WALSplitUtil(141): Moved hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting/jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789 to hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs 2023-07-14 19:03:31,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-07-14 19:03:31,868 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; SplitWALRemoteProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, worker=jenkins-hbase20.apache.org,38019,1689361393499 in 221 msec 2023-07-14 19:03:31,869 DEBUG [PEWorker-1] master.SplitWALManager(173): Release split WAL worker=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=17 2023-07-14 19:03:31,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=17, state=SUCCESS; SplitWALProcedure jenkins-hbase20.apache.org%2C34333%2C1689361388479.1689361389789, worker=jenkins-hbase20.apache.org,38019,1689361393499 in 231 msec 2023-07-14 19:03:31,876 INFO [PEWorker-2] master.SplitLogManager(171): hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/WALs/jenkins-hbase20.apache.org,34333,1689361388479-splitting dir is empty, no logs to split. 2023-07-14 19:03:31,876 INFO [PEWorker-2] master.SplitWALManager(106): jenkins-hbase20.apache.org,34333,1689361388479 WAL count=0, meta=false 2023-07-14 19:03:31,876 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(290): Check if jenkins-hbase20.apache.org,34333,1689361388479 WAL splitting is done? wals=0, meta=false 2023-07-14 19:03:31,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN}, {pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN}] 2023-07-14 19:03:31,880 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN 2023-07-14 19:03:31,880 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN 2023-07-14 19:03:31,881 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=24, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 19:03:31,881 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=25, ppid=17, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2023-07-14 19:03:31,882 DEBUG [jenkins-hbase20:38349] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-07-14 19:03:31,882 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,882 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c5039524108f6f5f0d888c4d54df6bc4, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:31,883 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361411882"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689361411882"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689361411882"}]},"ts":"1689361411882"} 2023-07-14 19:03:31,883 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689361411882"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1689361411882"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1689361411882"}]},"ts":"1689361411882"} 2023-07-14 19:03:31,884 WARN [RS-EventLoopGroup-4-2] ipc.NettyRpcConnection$2(294): Exception encountered while connecting to the server jenkins-hbase20.apache.org/148.251.75.209:34333 org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:34333 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155) at org.apache.hbase.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128) at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-07-14 19:03:31,885 DEBUG [RS-EventLoopGroup-4-2] ipc.FailedServers(52): Added failed server with address jenkins-hbase20.apache.org/148.251.75.209:34333 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: jenkins-hbase20.apache.org/148.251.75.209:34333 2023-07-14 19:03:31,973 DEBUG [hconnection-0x75745710-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:34333 this server is in the failed servers list 2023-07-14 19:03:31,988 DEBUG [PEWorker-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-07-14 19:03:31,989 INFO [RS-EventLoopGroup-6-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43360, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-07-14 19:03:31,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=24, state=RUNNABLE; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,38019,1689361393499}] 2023-07-14 19:03:31,993 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=25, state=RUNNABLE; OpenRegionProcedure c5039524108f6f5f0d888c4d54df6bc4, server=jenkins-hbase20.apache.org,38019,1689361393499}] 2023-07-14 19:03:32,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f8ff06c91c03cb8152ab689acbbf13c3, NAME => 't,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:32,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table t f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:32,150 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,151 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,153 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family f of region f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,154 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(130): Open hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c5039524108f6f5f0d888c4d54df6bc4, NAME => 'hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.', STARTKEY => '', ENDKEY => ''} 2023-07-14 19:03:32,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-07-14 19:03:32,155 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:32,155 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f 2023-07-14 19:03:32,155 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,156 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,156 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f8ff06c91c03cb8152ab689acbbf13c3 columnFamilyName f 2023-07-14 19:03:32,157 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,159 DEBUG [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info 2023-07-14 19:03:32,159 DEBUG [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info 2023-07-14 19:03:32,159 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c5039524108f6f5f0d888c4d54df6bc4 columnFamilyName info 2023-07-14 19:03:32,160 INFO [StoreOpener-c5039524108f6f5f0d888c4d54df6bc4-1] regionserver.HStore(310): Store=c5039524108f6f5f0d888c4d54df6bc4/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:32,162 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,165 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 1 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,166 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5276): Replaying edits from hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000005 2023-07-14 19:03:32,166 DEBUG [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(539): loaded hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/f/54c410394dc8478abfdf0a46bcfa0679 2023-07-14 19:03:32,167 INFO [StoreOpener-f8ff06c91c03cb8152ab689acbbf13c3-1] regionserver.HStore(310): Store=f8ff06c91c03cb8152ab689acbbf13c3/f, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-07-14 19:03:32,168 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,169 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5464): Applied 2, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000005 2023-07-14 19:03:32,170 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c5039524108f6f5f0d888c4d54df6bc4 1/1 column families, dataSize=78 B heapSize=488 B 2023-07-14 19:03:32,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=13 2023-07-14 19:03:32,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened f8ff06c91c03cb8152ab689acbbf13c3; next sequenceid=15; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72350661, jitterRate=0.07810886204242706}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:32,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:32,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3., pid=26, masterSystemTime=1689361412144 2023-07-14 19:03:32,181 DEBUG [hconnection-0x75745710-shared-pool-0] ipc.AbstractRpcClient(347): Not trying to connect to jenkins-hbase20.apache.org/148.251.75.209:34333 this server is in the failed servers list 2023-07-14 19:03:32,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,183 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f8ff06c91c03cb8152ab689acbbf13c3, regionState=OPEN, openSeqNum=15, regionLocation=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:32,183 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3.","families":{"info":[{"qualifier":"regioninfo","vlen":35,"tag":[],"timestamp":"1689361412183"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689361412183"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689361412183"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689361412183"}]},"ts":"1689361412183"} 2023-07-14 19:03:32,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=24 2023-07-14 19:03:32,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=24, state=SUCCESS; OpenRegionProcedure f8ff06c91c03cb8152ab689acbbf13c3, server=jenkins-hbase20.apache.org,38019,1689361393499 in 194 msec 2023-07-14 19:03:32,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=t, region=f8ff06c91c03cb8152ab689acbbf13c3, ASSIGN in 312 msec 2023-07-14 19:03:32,196 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/.tmp/info/b3189eab0749426abac0b2ae897a1d58 2023-07-14 19:03:32,205 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/.tmp/info/b3189eab0749426abac0b2ae897a1d58 as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info/b3189eab0749426abac0b2ae897a1d58 2023-07-14 19:03:32,213 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/info/b3189eab0749426abac0b2ae897a1d58, entries=2, sequenceid=5, filesize=4.8 K 2023-07-14 19:03:32,213 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for c5039524108f6f5f0d888c4d54df6bc4 in 43ms, sequenceid=5, compaction requested=false; wal=null 2023-07-14 19:03:32,214 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5086): Deleted recovered.edits file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/0000000000000000005 2023-07-14 19:03:32,218 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,220 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2023-07-14 19:03:32,221 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c5039524108f6f5f0d888c4d54df6bc4; next sequenceid=6; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72528279, jitterRate=0.0807555764913559}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-07-14 19:03:32,221 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c5039524108f6f5f0d888c4d54df6bc4: 2023-07-14 19:03:32,222 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2336): Post open deploy tasks for hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4., pid=27, masterSystemTime=1689361412144 2023-07-14 19:03:32,224 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2363): Finished post open deploy task for hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,225 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(158): Opened hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,225 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=c5039524108f6f5f0d888c4d54df6bc4, regionState=OPEN, openSeqNum=6, regionLocation=jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:32,226 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1689361412225"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1689361412225"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1689361412225"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1689361412225"}]},"ts":"1689361412225"} 2023-07-14 19:03:32,232 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=25 2023-07-14 19:03:32,232 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=25, state=SUCCESS; OpenRegionProcedure c5039524108f6f5f0d888c4d54df6bc4, server=jenkins-hbase20.apache.org,38019,1689361393499 in 237 msec 2023-07-14 19:03:32,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=17 2023-07-14 19:03:32,236 INFO [PEWorker-4] procedure.ServerCrashProcedure(251): removed crashed server jenkins-hbase20.apache.org,34333,1689361388479 after splitting done 2023-07-14 19:03:32,236 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=17, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=c5039524108f6f5f0d888c4d54df6bc4, ASSIGN in 354 msec 2023-07-14 19:03:32,236 DEBUG [PEWorker-4] master.DeadServer(114): Removed jenkins-hbase20.apache.org,34333,1689361388479 from processing; numProcessing=0 2023-07-14 19:03:32,239 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, state=SUCCESS; ServerCrashProcedure jenkins-hbase20.apache.org,34333,1689361388479, splitWal=true, meta=true in 1.3000 sec 2023-07-14 19:03:32,498 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e66a8ae to 127.0.0.1:51669 2023-07-14 19:03:32,498 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,499 INFO [Listener at localhost.localdomain/40851] client.TestScannerTimeout(124): END ************ test2772 2023-07-14 19:03:32,511 INFO [Listener at localhost.localdomain/40851] hbase.ResourceChecker(175): after: client.TestScannerTimeout#test2772 Thread=275 (was 293), OpenFileDescriptor=580 (was 571) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=122 (was 115) - SystemLoadAverage LEAK? -, ProcessCount=165 (was 165), AvailableMemoryMB=5195 (was 5200) 2023-07-14 19:03:32,511 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-07-14 19:03:32,511 INFO [Listener at localhost.localdomain/40851] client.ConnectionImplementation(1979): Closing master protocol: MasterService 2023-07-14 19:03:32,512 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x081026c2 to 127.0.0.1:51669 2023-07-14 19:03:32,512 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,512 DEBUG [Listener at localhost.localdomain/40851] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-07-14 19:03:32,512 DEBUG [Listener at localhost.localdomain/40851] util.JVMClusterUtil(257): Found active master hash=262601278, stopped=false 2023-07-14 19:03:32,512 INFO [Listener at localhost.localdomain/40851] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:32,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 19:03:32,517 INFO [Listener at localhost.localdomain/40851] procedure2.ProcedureExecutor(629): Stopping 2023-07-14 19:03:32,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-07-14 19:03:32,517 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:32,517 DEBUG [Listener at localhost.localdomain/40851] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d0880e4 to 127.0.0.1:51669 2023-07-14 19:03:32,517 DEBUG [Listener at localhost.localdomain/40851] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 19:03:32,517 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,38019,1689361393499' ***** 2023-07-14 19:03:32,517 INFO [Listener at localhost.localdomain/40851] regionserver.HRegionServer(2311): STOPPED: Shutdown requested 2023-07-14 19:03:32,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-07-14 19:03:32,518 INFO [RS:2;jenkins-hbase20:38019] regionserver.HeapMemoryManager(220): Stopping 2023-07-14 19:03:32,518 INFO [RS:2;jenkins-hbase20:38019] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-07-14 19:03:32,518 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-07-14 19:03:32,518 INFO [RS:2;jenkins-hbase20:38019] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-07-14 19:03:32,518 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(3305): Received CLOSE for c5039524108f6f5f0d888c4d54df6bc4 2023-07-14 19:03:32,518 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(3305): Received CLOSE for f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,519 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:32,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c5039524108f6f5f0d888c4d54df6bc4, disabling compactions & flushes 2023-07-14 19:03:32,519 DEBUG [RS:2;jenkins-hbase20:38019] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fa40606 to 127.0.0.1:51669 2023-07-14 19:03:32,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,519 DEBUG [RS:2;jenkins-hbase20:38019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,519 INFO [RS:2;jenkins-hbase20:38019] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-07-14 19:03:32,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. after waiting 0 ms 2023-07-14 19:03:32,520 INFO [RS:2;jenkins-hbase20:38019] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-07-14 19:03:32,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,520 INFO [RS:2;jenkins-hbase20:38019] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-07-14 19:03:32,520 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(3305): Received CLOSE for 1588230740 2023-07-14 19:03:32,520 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1474): Waiting on 3 regions to close 2023-07-14 19:03:32,520 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1478): Online Regions={c5039524108f6f5f0d888c4d54df6bc4=hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4., f8ff06c91c03cb8152ab689acbbf13c3=t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3., 1588230740=hbase:meta,,1.1588230740} 2023-07-14 19:03:32,520 DEBUG [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1504): Waiting on 1588230740, c5039524108f6f5f0d888c4d54df6bc4, f8ff06c91c03cb8152ab689acbbf13c3 2023-07-14 19:03:32,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-07-14 19:03:32,521 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-07-14 19:03:32,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-07-14 19:03:32,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-07-14 19:03:32,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-07-14 19:03:32,521 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=1.75 KB heapSize=3.68 KB 2023-07-14 19:03:32,526 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/namespace/c5039524108f6f5f0d888c4d54df6bc4/recovered.edits/8.seqid, newMaxSeqId=8, maxSeqId=5 2023-07-14 19:03:32,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c5039524108f6f5f0d888c4d54df6bc4: 2023-07-14 19:03:32,527 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1689361390656.c5039524108f6f5f0d888c4d54df6bc4. 2023-07-14 19:03:32,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing f8ff06c91c03cb8152ab689acbbf13c3, disabling compactions & flushes 2023-07-14 19:03:32,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. after waiting 0 ms 2023-07-14 19:03:32,528 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,535 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/default/t/f8ff06c91c03cb8152ab689acbbf13c3/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=14 2023-07-14 19:03:32,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for f8ff06c91c03cb8152ab689acbbf13c3: 2023-07-14 19:03:32,536 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed t,,1689361391507.f8ff06c91c03cb8152ab689acbbf13c3. 2023-07-14 19:03:32,553 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.75 KB at sequenceid=22 (bloomFilter=false), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/info/2f80c66684a743b9b15ffd847a9f6cca 2023-07-14 19:03:32,561 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/.tmp/info/2f80c66684a743b9b15ffd847a9f6cca as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info/2f80c66684a743b9b15ffd847a9f6cca 2023-07-14 19:03:32,567 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/info/2f80c66684a743b9b15ffd847a9f6cca, entries=16, sequenceid=22, filesize=6.4 K 2023-07-14 19:03:32,568 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~1.75 KB/1796, heapSize ~3.16 KB/3240, currentSize=0 B/0 for 1588230740 in 47ms, sequenceid=22, compaction requested=false 2023-07-14 19:03:32,574 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-07-14 19:03:32,574 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-07-14 19:03:32,575 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:32,576 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/data/hbase/meta/1588230740/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=15 2023-07-14 19:03:32,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-07-14 19:03:32,578 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-07-14 19:03:32,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-07-14 19:03:32,578 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-07-14 19:03:32,720 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,38019,1689361393499; all regions closed. 2023-07-14 19:03:32,734 DEBUG [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs 2023-07-14 19:03:32,734 INFO [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38019%2C1689361393499.meta:.meta(num 1689361411494) 2023-07-14 19:03:32,740 DEBUG [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/oldWALs 2023-07-14 19:03:32,740 INFO [RS:2;jenkins-hbase20:38019] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C38019%2C1689361393499:(num 1689361393711) 2023-07-14 19:03:32,740 DEBUG [RS:2;jenkins-hbase20:38019] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,741 INFO [RS:2;jenkins-hbase20:38019] regionserver.LeaseManager(133): Closed leases 2023-07-14 19:03:32,741 INFO [RS:2;jenkins-hbase20:38019] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-07-14 19:03:32,741 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 19:03:32,742 INFO [RS:2;jenkins-hbase20:38019] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:38019 2023-07-14 19:03:32,743 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,38019,1689361393499 2023-07-14 19:03:32,743 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:32,743 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-07-14 19:03:32,744 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,38019,1689361393499] 2023-07-14 19:03:32,744 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,38019,1689361393499; numProcessing=1 2023-07-14 19:03:32,745 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,38019,1689361393499 already deleted, retry=false 2023-07-14 19:03:32,745 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,38019,1689361393499 expired; onlineServers=0 2023-07-14 19:03:32,745 INFO [RegionServerTracker-0] regionserver.HRegionServer(2297): ***** STOPPING region server 'jenkins-hbase20.apache.org,38349,1689361387512' ***** 2023-07-14 19:03:32,745 INFO [RegionServerTracker-0] regionserver.HRegionServer(2311): STOPPED: Cluster shutdown set; onlineServer=0 2023-07-14 19:03:32,746 DEBUG [M:0;jenkins-hbase20:38349] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e9bf005, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-07-14 19:03:32,746 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegionServer(1144): stopping server jenkins-hbase20.apache.org,38349,1689361387512 2023-07-14 19:03:32,746 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegionServer(1170): stopping server jenkins-hbase20.apache.org,38349,1689361387512; all regions closed. 2023-07-14 19:03:32,746 DEBUG [M:0;jenkins-hbase20:38349] ipc.AbstractRpcClient(494): Stopping rpc client 2023-07-14 19:03:32,746 DEBUG [M:0;jenkins-hbase20:38349] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-07-14 19:03:32,746 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-07-14 19:03:32,746 DEBUG [M:0;jenkins-hbase20:38349] cleaner.HFileCleaner(317): Stopping file delete threads 2023-07-14 19:03:32,746 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689361389487] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1689361389487,5,FailOnTimeoutGroup] 2023-07-14 19:03:32,746 INFO [M:0;jenkins-hbase20:38349] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-07-14 19:03:32,746 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689361389487] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1689361389487,5,FailOnTimeoutGroup] 2023-07-14 19:03:32,747 INFO [M:0;jenkins-hbase20:38349] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-07-14 19:03:32,747 INFO [M:0;jenkins-hbase20:38349] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-07-14 19:03:32,747 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-07-14 19:03:32,747 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-07-14 19:03:32,747 DEBUG [M:0;jenkins-hbase20:38349] master.HMaster(1512): Stopping service threads 2023-07-14 19:03:32,747 INFO [M:0;jenkins-hbase20:38349] procedure2.RemoteProcedureDispatcher(119): Stopping procedure remote dispatcher 2023-07-14 19:03:32,747 ERROR [M:0;jenkins-hbase20:38349] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-07-14 19:03:32,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[IPC Client (902248133) connection to localhost.localdomain/127.0.0.1:40151 from jenkins,5,PEWorkerGroup] 2023-07-14 19:03:32,748 INFO [M:0;jenkins-hbase20:38349] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-07-14 19:03:32,748 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-07-14 19:03:32,748 DEBUG [M:0;jenkins-hbase20:38349] zookeeper.ZKUtil(398): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-07-14 19:03:32,749 WARN [M:0;jenkins-hbase20:38349] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-07-14 19:03:32,749 INFO [M:0;jenkins-hbase20:38349] assignment.AssignmentManager(315): Stopping assignment manager 2023-07-14 19:03:32,749 INFO [M:0;jenkins-hbase20:38349] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-07-14 19:03:32,749 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-07-14 19:03:32,749 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:32,749 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:32,749 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-07-14 19:03:32,749 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:32,749 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=91.63 KB heapSize=110.36 KB 2023-07-14 19:03:32,766 INFO [M:0;jenkins-hbase20:38349] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=91.63 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7d97f935109443b4934082ded6bc5a3f 2023-07-14 19:03:32,774 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/7d97f935109443b4934082ded6bc5a3f as hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7d97f935109443b4934082ded6bc5a3f 2023-07-14 19:03:32,781 INFO [M:0;jenkins-hbase20:38349] regionserver.HStore(1080): Added hdfs://localhost.localdomain:40151/user/jenkins/test-data/f11d3a9c-9bcf-5866-817e-31af5269344b/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/7d97f935109443b4934082ded6bc5a3f, entries=27, sequenceid=212, filesize=8.2 K 2023-07-14 19:03:32,782 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegion(2948): Finished flush of dataSize ~91.63 KB/93824, heapSize ~110.34 KB/112992, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 33ms, sequenceid=212, compaction requested=false 2023-07-14 19:03:32,784 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-07-14 19:03:32,784 DEBUG [M:0;jenkins-hbase20:38349] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-07-14 19:03:32,787 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-07-14 19:03:32,788 INFO [M:0;jenkins-hbase20:38349] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-07-14 19:03:32,788 INFO [M:0;jenkins-hbase20:38349] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:38349 2023-07-14 19:03:32,789 DEBUG [M:0;jenkins-hbase20:38349] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,38349,1689361387512 already deleted, retry=false 2023-07-14 19:03:33,003 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:33,003 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): master:38349-0x1008cdd89820000, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:33,003 INFO [M:0;jenkins-hbase20:38349] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,38349,1689361387512; zookeeper connection closed. 2023-07-14 19:03:33,103 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:33,104 DEBUG [Listener at localhost.localdomain/40851-EventThread] zookeeper.ZKWatcher(600): regionserver:38019-0x1008cdd89820008, quorum=127.0.0.1:51669, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-07-14 19:03:33,103 INFO [RS:2;jenkins-hbase20:38019] regionserver.HRegionServer(1227): Exiting; stopping=jenkins-hbase20.apache.org,38019,1689361393499; zookeeper connection closed. 2023-07-14 19:03:33,104 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@617925bb] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@617925bb 2023-07-14 19:03:33,105 INFO [Listener at localhost.localdomain/40851] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-07-14 19:03:33,105 WARN [Listener at localhost.localdomain/40851] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 19:03:33,109 INFO [Listener at localhost.localdomain/40851] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 19:03:33,221 WARN [BP-74110187-148.251.75.209-1689361384905 heartbeating to localhost.localdomain/127.0.0.1:40151] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 19:03:33,221 WARN [BP-74110187-148.251.75.209-1689361384905 heartbeating to localhost.localdomain/127.0.0.1:40151] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-74110187-148.251.75.209-1689361384905 (Datanode Uuid 59942dd0-ebfc-489d-ae82-eba6e968d81c) service to localhost.localdomain/127.0.0.1:40151 2023-07-14 19:03:33,222 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data3/current/BP-74110187-148.251.75.209-1689361384905] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 19:03:33,223 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data4/current/BP-74110187-148.251.75.209-1689361384905] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 19:03:33,224 WARN [Listener at localhost.localdomain/40851] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-07-14 19:03:33,225 INFO [Listener at localhost.localdomain/40851] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-07-14 19:03:33,334 WARN [BP-74110187-148.251.75.209-1689361384905 heartbeating to localhost.localdomain/127.0.0.1:40151] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-07-14 19:03:33,334 WARN [BP-74110187-148.251.75.209-1689361384905 heartbeating to localhost.localdomain/127.0.0.1:40151] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-74110187-148.251.75.209-1689361384905 (Datanode Uuid e84c29ef-a447-41f7-afb3-6846372b9cea) service to localhost.localdomain/127.0.0.1:40151 2023-07-14 19:03:33,334 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data1/current/BP-74110187-148.251.75.209-1689361384905] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 19:03:33,335 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/14b72a4a-e98a-65fd-3dcd-dca0ecc533cc/cluster_18683e70-3007-4a6f-066e-042e3f688ae7/dfs/data/data2/current/BP-74110187-148.251.75.209-1689361384905] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-07-14 19:03:33,369 INFO [Listener at localhost.localdomain/40851] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-07-14 19:03:33,487 INFO [Listener at localhost.localdomain/40851] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-07-14 19:03:33,518 INFO [Listener at localhost.localdomain/40851] hbase.HBaseTestingUtility(1293): Minicluster is down